Before I ever became a computational humanist, I was a James Joyce scholar. What drew me to Joyce was not difficulty for its own sake, but his extraordinary intimacy with language and his willingness to follow the word wherever it led, even when that path violated every convention of narrative efficiency or reader ease. A book like Ulysses is not simply stylistically distinctive; it is a novel that reinvents its style chapter by chapter, adopting and discarding “the rules” with confidence. Finnegans Wake goes even further by dissolving the boundaries between languages, voices, and even consciousness. It is difficult to imagine either book emerging from a generative system optimized to reproduce learned patterns. The same could be said of Mrs Dalloway, whose compression of time and interiority also helped to reshape the modern novel. These books endure not because they conform to recognizable patterns, but because they violate them, because their authors chose risk, dare I say “art,” over guarantees and safety.

I begin here, with what are arguably two extreme examples, because the current moment poses a consequential question for writers and for publishing more broadly: what happens when the dominant tools of narrative production are designed not to defy patterns, but to reproduce them?

Writers are not wrong to feel tempted by generative AI. I’ll confess, along with Oscar Wilde, to being able to resist anything but temptation. For the first time in history, I can sit down with a machine and within a matter of a few minutes and some very generic prompting produce something that looks like a story: complete sentences, plausible characters, and recognizable plot twists. It can draft scenes on demand, mimic familiar voices and styles, including my own(!), and even produce something that feels, at least at a glance, coherent and competent. For a profession built on long hours of solitude and uncertainty (Joyce spent seven years on Ulysses and seventeen on the Wake), the allure is obvious, especially if the goal is commercial success (i.e not Joyce and Woolf).

But that allure deserves careful scrutiny.

Long before generative AI could whip me up a paragraph of “once upon a time” quality fiction, I spent years studying stories from the outside. I measured them, I mapped them, and I tried to understand at a very deep level why some narratives move readers while others do not. In Macroanalysis and later in The Bestseller Code, I looked at thousands of novels at once, searching for patterns: recurring emotional rhythms, thematic trajectories, and stylistic regularities that seemed to correlate with reader engagement and commercial success. I was not trying to explain how to write a novel so much as how novels, collectively, tend to work.

One of the most persistent things discussed in literary studies is that stories tend to follow recognizable patterns. This insight is not new. Gustav Freytag sketched his famous pyramid in the nineteenth century. Aristotle was already thinking in similar terms two thousand years earlier. T.S Elliot, Joseph Campbell, Harold Bloom (Anxiety of Influence) and many more have written about “standing on the shoulders of giants.” What computation added to this observation was scale. Instead of arguing from a handful of canonical texts, we could observe patterns across thousands of books.

Using tools like my Syuzhet package—built with the fairly blunt instrument of a sentiment lexicon—we can trace rises and falls in emotional valence across a narrative. While the approach is far from perfect, it becomes surprisingly powerful at scale. Zoom out far enough and noise gives way to pattern. Certain shapes recur again and again. Some stories rise steadily and crash. Others fall before recovering. Still others oscillate, keeping readers emotionally off balance until the final pages. 

In The Bestseller Code, Jodie Archer and I showed that some of these shapes appear more frequently in commercially successful novels than in the literary field as a whole. Stories have shapes; writers have styles; literature is filled with recurring topics and themes. We can measure all of them. That measuring work we call analysis, and it’s a type of patten recognition.  And this work was always descriptive, not prescriptive. But description has a way of becoming prescription once it enters the marketplace.

Writing is not pattern recognition. Writing is decision-making. It is choosing this word rather than a dozen other equally plausible alternatives for reasons that are often emotional, ethical, idiosyncratic, or artistic, but never probabilistic. ¹ Do we really want to farm out that decision making to an AI?  Large language models are trained on massive corpora of existing text, including fiction. They learn statistical regularities at many levels simultaneously: word choice, syntax, pacing, dialogue patterns, and even emotional progression. In effect, they internalize the same kinds of narrative patterns that computational critics measure from the outside. When such black box systems generate stories, however, they are not inventing narrative logic from scratch. They are recombining learned structures that already reflect the bias and gravitational pull of publishing markets, reader expectations, and historical convention.

This is why, at least in my experience, AI-generated fiction so often feels competent but hollow. The emotional beats arrive when expected. The tension rises on cue. The resolution lands where it should. In my experiments, at least, the result is rather cliché. Generative AI excels at producing smooth, competent sounding prose. It avoids sharp edges. It satisfies expectations. It is, by design, predictable pabulum.

And pabulum provides a useful analogy.

Highly processed food is optimized for consistency, mass appeal and predictability. Generative AI excels at producing the literary equivalent of processed food. It’s not necessarily bad, but it might not be especially good. Like processed food, AI can be used thoughtfully, but it probably should not become the default engine of narrative production. Organic food, on the other hand, is uneven. It (often) costs more. It risks failure and, for many, it feeds not just the body, but the soul.

Publishing already lives with a tension between art and market. Editors talk about “the book that feels familiar but different enough.” Agents see waves of submissions chasing last year’s success. Writers feel pressure, explicit or not, to align with what sells. Generative AI threatens to intensify this pressure by making imitation frictionless.

If models are trained primarily on successful books, and if writers increasingly rely on those models for drafting, ideation, or revision, the system begins to feed on itself. Patterns reinforce patterns. Stylistic risk narrows. Innovation becomes statistically improbable because the ratio of organic to processed prose is shrinking. The danger here is not that AI will replace writers. It is that it will quietly reshape what counts as acceptable prose, nudging the literary ecosystem toward a smoother, safer, more predictable center. Pabulum.

Despite working at the intersection of computation and culture for much of my career, I have been deliberately conservative when it comes to using AI to generate prose. It’s damn fun to play with, and when I need a quick limerick or some drivel for a greeting card, it gets the job done.  Once upon a time, during an existential phase, I published a short story. It was pretty good, but my interest has always been analytical rather than generative: I like using machines to see what humans, reading one book at a time, cannot easily see.

Computational tools, including large language models, can be extraordinarily good at revealing patterns, surfacing blind spots, and challenging our intuitions about narrative. They can help writers understand pacing, emotional balance, and reader response at a macro level. Used this way, they expand awareness rather than replace judgment.

So in this context, generative AI can be used thoughtfully as a diagnostic tool, a brainstorming partner, or a way to explore alternatives. But I do worry about a future in which the default engine of narrative production is the linguistic equivalent of processed food. The more we outsource first drafts, emotional scaffolding, and stylistic decision-making to the food processor of prose production, the more we train ourselves to accept work that is technically proficient but flavorless. Organic writing takes longer. It risks failure, and it is the probably the only kind of writing that has ever changed how we see the world. As tempting as synthetic stories may be, we should be careful what we make habitual. Eventually, we become accustomed to what we consume, and we are what we eat.

1. Even if we accept Robert Sapolsky’s (compelling) argument that there is no free will, and the natural extension that human beings are just probabilistic word generation machines, human being are still unique, individual word generators and not a hive mind of all that has been ever written. ↩︎

I recently reread David Epstein’s Range (2019), a book I first encountered a few years ago when it seemed every leadership forum was extolling the virtues of grit, 10,000 hours, and early specialization. Epstein pushed back, persuasively arguing that generalists, not specialists, are better equipped to solve complex problems, especially in domains where rules are unclear and outcomes are unpredictable. His thesis struck me as a welcome corrective and a fitting principle for the Dean of a College of Arts and Sciences (which I was at the time) to embrace. Reading it again now, in the post–generative AI world, I find it more than just persuasive; I find it essential.

Epstein’s central claim is that those who explore broadly, delay specialization, and learn through analogy and synthesis are better prepared for the “wicked” problems of the world—problems that don’t come with tidy instructions or immediate feedback. That idea was always relevant. But as generative AI takes on more and more of the tasks traditionally associated with specialized expertise (e.g. software programming, legal research, medical diagnostics, financial analysis, language translation, writing etc.), Epstein’s argument takes on new urgency. We need a different kind of pedagogy now, one that privileges range, depth, and judgment over memorization and narrow skill-building.

Memorization Is Obsolete. But Thinking Isn’t.

Let’s be honest…if you want fast facts, crisp summaries, or a list of references, a large language model can do the job faster and more reliably than most humans. The days when being able to recall information conferred professional advantage are behind us. But here’s the rub: what AI cannot do, at least not yet, is to make meaningful analogies across domains, or to recognize when a familiar pattern no longer applies (AI clings to statistical priors and probabilities), or to ask truly generative questions, which is to say questions that open new avenues of inquiry rather than simply remixing what’s already known.

Those capacities are learned not through repetition or drill, but through what Epstein calls “sampling”: through exposure to different ways of thinking, working, and seeing. This is precisely what a traditional liberal arts education aimed to foster. In fact, I’d argue that the habits of mind developed through broad study of mathematics and science, but also of literature, history, philosophy, and the arts (disciplines too often marginalized in the STEM-obsessed discourse) are exactly what we need to cultivate in students if we want them to thrive alongside AI. And I say this as someone who has invested heavily in STEM, both personally and professionally.

The more things change the more they stay the same

When my father was considering college, a liberal arts major was seen as the doorway to anything. Higher Ed was still a rather elite pursuit: medicine, teaching, and law were represented to him as “respectable” pursuits, but, then again, so was classics. He majored in English and math and felt well-prepared for a variety of roles in business. By the time I was in high school, the conventional wisdom about a broad foundation had shifted.  My father still valued his liberal arts foundation, but he advised me to specialize and pursue finance, accounting, business, or possibly law.  I did not, but I was convinced I needed a “practical” degree and spent one year as an architecture student before decamping to the liberal arts and an English major with several minors.  

By the time I was graduating from college, the conventional advice was shifting again, this time in a big way toward computer science and engineering.  I caught that wave and became an “early adopter” of programming, but mine was intended as a hobbyist’s pursuit, definitely not a career. Or so I thought. 

By the 2010s, CS and engineering had broadened to anything STEM, and quantitative degrees were touted as the surefire and sensible choice for job security in the modern world. Healthcare, especially “Pre-Med,” was an emerging area of attention and received honorable mention. Meanwhile the edusphere was rife with jokes about the most effective way to get an English major’s attention: just yell “waiter.”

The pendulum of conventional wisdom swung wide in the direction of increased specialization. Computer science and engineering came to dominate the conversation.  But soon a problem surfaced.  Higher ed was producing a lot of experts, but these experts weren’t very well rounded.  In 2019, the Wall Street Journal profiled how Northeastern University began requiring CS majors to take theater classes (specifically improv!) in order to “sharpen uniquely human skills” and build “empathy, creativity and teamwork [as a] competitive advantage over machines in the era of artificial intelligence.”¹  Prescient?

Wicked Learning Environments Are the Norm Now

Epstein draws a sharp contrast between “kind” environments (like chess or golf) where patterns repeat and feedback is immediate, and “wicked” environments where feedback is sparse, misleading, or delayed. The world of work is not kind, and the world of generative AI is wicked in spades. These models are probabilistic, opaque, and massively influential. They’re already reshaping industries and knowledge work, and their decisions are often unexplainable—even to their creators.

Navigating this world demands not just technical fluency, but epistemic humility and conceptual agility. It requires the ability to think critically about systems, to understand where they might go wrong, and to imagine alternative futures. These are not traits we cultivate by marching students through test prep or narrow curricula. They’re cultivated through play, analogy, experimentation, and yes, through wandering widely around the course catalog and thinking deeply.

AI Is a Specialist. We Can Be Generalists.

Ironically, the very thing that makes AI powerful, especially when the models are fine-tuned to a particular task or adapted for a specific domain, is also a potential blind spot. Generative models are trained on what already exists. They can remix, but they can’t reimagine, not really. They can simulate reasoning, but they don’t have perspective. They can write beautifully fluent text, but they don’t have skin in the game or any real sense of how the words on the page convey meaning(s). That’s our job.

In a recent article for The Atlantic, Matteo Wong recounts a conversation with an AI researcher who was “rethinking the value of school.”² Wong writes: “One entrepreneur told me that today’s bots may already be more scholastically capable than his teenage son will ever be, leading him to doubt the value of a traditional education.”  I can’t help wondering what that entrepreneur was thinking when using the word “traditional.”

If anything, the rise of generative AI reopens space for the (very traditional) Renaissance mind, for thinkers who can roam across domains, connect unlikely dots, and bring ethical insight to technical problems. The human edge isn’t in being faster or more encyclopedic or more “scholastically capable”; it’s in being wiser. That’s a distinctly generalist strength.

Toward a Post-AI Pedagogy

So what does this mean for teaching and learning? Arguably, it means we need to stop confusing learning with content acquisition and specialization. When it comes to content acquisition, the AIs will beat us every time. This means doubling down on slow learning, on open-ended inquiry, on the value of taking time to understand why something matters, not just how to do it. It means encouraging students to read outside their major, to embrace intellectual detours, and to reflect on what they know and don’t know. 

To be clear, I’m not suggesting we abandon technical training or STEM. But we need to reframe its purpose. In a world where tools evolve faster than syllabi, the lasting value of higher education lies not in tool mastery but in the transferability of judgment, in the ability to reason analogically and ethically under conditions of uncertainty. 

Reading Range again has reminded me that the best preparation for a world shaped by AI might not be more AI—but more humanity. More slow thinking, more curiosity, and more range.

1. Castellanos, Sara. “‘Oh, My God, Where Is This Going?’ When Computer-Science Majors Take Improv.” Wall Street Journal. May 14, 2019. (https://www.wsj.com/articles/oh-my-god-where-is-this-going-when-computer-science-majors-take-improv-11557846729)
2. Wong, Matteo. “The AI Industry is Radicalizing.” The Atlantic. July 8, 2025. (https://www.theatlantic.com/technology/archive/2025/07/ai-radicalization-civil-war/683460/)

On Friday I posted an updated version of Syuzhet (1.0.4) to CRAN. This version has been available over on GitHub for a while now. In version 1.0.4, support for sentiment detection in several languages was added by using the expanded NRC lexicon from Saif Mohammed. The lexicon includes sentiment values for 13,901 words in each of the following languages: Arabic, Basque, Bengali, Catalan, Chinese_simplified, Chinese_traditional, Danish, Dutch, English, Esperanto, Finnish, French, German, Greek, Gujarati, Hebrew, Hindi, Irish, Italian, Japanese, Latin, Marathi, Persian, Portuguese, Romanian, Russian, Somali, Spanish, Sudanese, Swahili, Swedish, Tamil, Telugu, Thai, Turkish, Ukranian, Urdu, Vietnamese, Welsh, Yiddish, Zulu.

At the time of this release, however, Syuzhet will only work with languages that use Latin character sets. This effectively means that “Arabic”, “Bengali”, “Chinese_simplified”, “Chinese_traditional”, “Greek”, “Gujarati”, “Hebrew”, “Hindi”, “Japanese”, “Marathi”, “Persian”, “Russian”, “Tamil”, “Telugu”, “Thai”, “Ukranian”, “Urdu”, “Yiddish” are not supported even though these languages are part of the extended NRC dictionary and can be accessed via the get_sentiment_dictionary() function. I have heard from several of my non-English native speaking students and a few others on Twitter that the German, French, and Spanish results seem to be good. Your mileage may vary. For details on the lexicon, see NRC Emotion Lexicon.

Also in this release is support for user created lexicons. To work, users create their own custom lexicon as a data frame with at least two columns named “word” and “value.” Here is a simplified example:

my_text <- "I love when I see something beautiful. I hate it when ugly feelings creep into my head."
char_v <- get_sentences(my_text)
custom_lexicon <- data.frame(word=c("love", "hate", "beautiful", "ugly"), value=c(1,-1,1, -1))
method <- "custom"
my_custom_values <- get_sentiment(char_v, method = method, lexicon = custom_lexicon)
my_custom_values

With contributions from Philip Bulsink, support for parallel processing was added so that one can call get_sentiment() and provide cluster information from parallel::makeCluster() to achieve results quicker on systems with multiple cores.  

Thanks also to Jennifer Isasi, Tyler Rinker, “amrrs,” and Oliver Keyes for recent suggestions/contributions/QA.

Examples of how to use these new functions and languages are in the updated vignette.

Back in December I posted results from a human validation experiment in which machine extracted sentiment values were compared to human coded values. The results were encouraging. In the spring, we mined the human coded sentences to help create a new sentiment dictionary that would, in theory, be more sensitive to the sort of sentiment words common to fiction (whereas existing sentiment dictionaries tend to be derived from movie and/or product review corpora). This dictionary was implemented as the default in the latest release of the Syuzhet R package (2016-04-28).

Over the summer, a new group of six human-coders was hired to read novels and score the sentiment of every sentence. Each novel was read by three human-coders. In the graphs that follow below, a simple moving average is used to plot the mean sentiment of the three students (black line) along side the values derived from the new “Syuzhet” dictionary (red line). Each graph reports the Pearson product-moment correlation coefficient.

This fall we will continue gathering human data by reading additional books. Once we have a few more books read, we’ll post a more detailed report, including data about inter-coder agreement and which machine methods produced results closest to the humans.

Ben Schmidt’s and Scott Enderle’s recent entries into the syuzhet discussion have beaten the last of the low pass filter out of me. I’m not entirely ready to concede that Fourier is useless for the larger problem, but they have convinced me that a better solution than the low pass is possible and probably warranted. What that better solution is remains an open question, but Ben has given us some things to consider.

In a nutshell, there were two essential elements to Vonnegut’s challenge that the low pass method seemed to be solving.  According to Vonnegut, this business of story shape “is an exercise in relativity” in which “it is the shape of the curves that matter and not their point of origin.”  Vonnegut imagined a system of plot in which the high and lows of good fortune and ill fortune are internally relative.  In this way, a very negative book such as Blood Meridian will have an absolute high and an absolute low that can be compared to another book that, though more positive on a whole, will also have an absolute high and an absolute low. The object of analysis is not the degree of positive or negative valence but the location of the spikes and troughs of that valence relative to the beginning and end of the book.  When conceived of in these terms, the ringing artifacts of the low pass filter seem rather trivial because the objective was not to perfectly represent the valence but to dramatize the shifts in valence.

As Ben has pointed out, however, the edges of the Fourier method present a different sort of problem; they assume that story plots are periodic, repeating signals.  The problem, as Ben puts it, is that the method “imposes an assumption that the start of [a] plot lines up with the end of a plot.”

Over the weekend, Ben and I exchanged a few emails, and I acknowledged that I had been overlooking these edge distortions in favor of a big picture perspective of the general shape.  Some amount of distortion, after all, must be tolerated if we want to produce a smooth shape.  As Israel Arroyo pointed out in a tweet, “endpoints are problematic in most smoothers and filters.”  With a simple rolling window, for example, the averaging can’t start until we are already half the distance of the window into the sequence.  Figure 1, which shows four options for smoothing Portrait of the Artist, highlights the moving average problem in blue.[1]

Figure 1

Looking only at figure one, it would be hard to argue against Fourier as a beautiful representation of the plot shape.  Figure 2 shows the same four methods applied to Dorian Gray.  Here again, the Fourier method seems to provide a fair representation.  In this case, however, we begin to see a problem forming at the end of the book.  The red lowess line is trending down while the green Fourier is reaching up in order to complete its cycle.  The beginning still looks good, and perhaps the distortion at the end can be tolerated, but it’s certainly not ideal.

Figure 2

Unfortunately, some sentiment trajectories appear to create a far more pronounced problem.  At Ben’s suggestion, I ran the same experiments with Madame Bovary.  The resulting plot is shown in figure 3.  I’ve not read Bovary in many years, so I can’t recall too many details about plot, but I do remember that it does not end well for anyone.  The shape of the green Fourier line at the end of figure 3, however, suggests some sort of uptick in positive sentiment that I suspect is not present in the text. The start of the shape, on the left, also looks problematic compared to the other smoothers.

Figure 3

With the first two figures, I think a case can be made that the Fourier line offers a fair representation of the emotional trajectory.  Making such a case for Bovary is not inconceivable if we ignore the edges, but it is clearly a stretch, and there is no denying that the lowess smoother does a better job.

In our email exchange about these different options, Ben included a graphic showing how various methods model four different books.  At least in these examples, loess (fifth row of figure 4) appears to be the top contender if we seek a representation that is both maximally smooth and maximally approximate.

Figure 4

In order to fully solve Vonnegut’s challenge, an alternative to percentage chunking is still necessary.  Longer segments in longer books will tend toward a neutral valence.  Figuring that out is work for the future.  For now, the Bovary example provides precisely the sort of validation/invalidation I was hoping to elicit by putting the package online.

RIP low-pass filter.[2]

FOOTNOTES:

[1] There are some more elegant ways to deal with filling in the flat edges, but keeping it simple here for illustration.

[2] I’m grateful to everyone who has engaged in this discussion, especially Annie Swafford, Daniel Lepage, Ted Underwood, Andrew Piper, David Bamman, Scott Enderle, and Ben Schmidt.  It has been a very engaging couple of weeks, and along the way I could not help but think of what this discussion might have looked like in print: it would have taken years to unfold!  Despite some emotional high and lows of its own, this has been a productive exercise and a great example of how valuable open code and the digital commons can be for progress.

Introduction

This post is a followup to A Novel Method for Detecting Plot posted June 15, 2014.

For the past few years, I have been exploring the relationship between sentiment and plot shape in fiction. Earlier today I posted an R package titled “syuzhet” to github. The package is designed to extract sentiment and plot information from prose. Methods for text import, sentiment extraction, and plot arc modeling are described in the documentation and in the package vignette. What follows below is a blog-friendly version of a longer academic paper describing how I employed this package to study plot in a corpus of ~50,000 novels.

Background

When I began the research that lead to this package, my goal was to study positive and negative emotions in literature across time, much in the same way that I had studied style and theme in Macroanalysis. Along the way, however, I discovered that fluctuations in sentiment can serve as a rather natural proxy for fluctuations in plot movement. Studying plot shifts via sentiment analysis turned out to be a far more interesting project than the simple study of sentiment, and my research got a huge boost when I stumbled upon a video of Kurt Vonnegut describing plot in precisely these terms.

After seeing the video and hearing Vonnegut’s opening challenge (“There’s no reason why the simple shapes of stories can’t be fed into computers”), I set out to develop a systematic way of extracting plot arcs from fiction. I felt this might help me to better understand and visualize how narrative is constructed. The fundamental idea, of course, was nothing new. What I was after is what the Russian formalist Vladimir Propp had defined as the narrative’s syuzhet (the organization of the narrative) as opposed to its fabula (raw elements of the story).

Syuzhet is concerned with the linear progression of narrative from beginning (first page) to the end (last page), whereas fabula is concerned with the specific events of a story, events which may or may not be related in chronological order. When we study fabula, which is what we typically do in literature courses, we mentally reconstruct the events into chronological order. We hope that this reconstruction of the fabula will help us understand the experience of the characters, the core story, etc. When we study the syuzhet, we are not so much concerned with the order of the fictional events but specifically interested in the manner in which the author presents those events to readers.

Consider the technique that radio personality Paul Harvey used in his iconic radio show “The Rest of the Story.” In each story, Harvey would hold back certain key elements until the very end of the program. The narrative would appear to have reached its conclusion, and then Harvey would say, “and now, the rest of the story.” At this point, he would reveal the held back information and the listener would reconstruct the entire fabula. The effect (and affect) of Harvey’s technique, the syuzhet, was usually stunning and pleasantly surprising. Had the story been told in simple chronological order, it would have been bland, perhaps even boring. What gave Harvey’s show power was his narrative technique.

This power was largely derived from the organization of the narrative elements and the manner in which Harvey offered listeners clues and then used narrative and language to evoke both curiosity and emotional response. What Harvey said and how he said it, were critical elements to the overall effect of the story. Harvey’s success was in finding and mastering a particular style of plot, a plot that has much in common with those found in mystery and detective fiction. A series of clues is presented along side a series of misdirections and the mystery is ultimately resolved in some grand reveal that defies expectations.

A Finite Number of Plots

But this Harvey method is just one among many possible plots. Countless scholars and non-scholars have pontificated about the possibility of a finite set of fundamental or archetypal plot shapes.

One of the more recent and famous/infamous of these scholars is Christopher Booker, whose 2004 book, titled The Seven Basic Plots: Why We Tell Stories, argues for a Jungian inspired understanding of plot in terms of seven basic archetypes. Booker’s work appears to be strongly influenced by prior work describing plot in terms of conflict. These core conflicts will be familiar to students of literature: such constructions were once taught to us under the headings of “man vs. man,” “man against nature,” “man vs. society,” and so on.

Other scholars have offered other numbers. William Foster-Harris has argued in favor of three basic patterns of plot The Basic Patterns of Plot (Foster-Harris. University of Oklahoma Press, 1959.); Ronald B. Tobias has argued for twenty (Tobias, Ronald B. 20 Master Plots. Cincinnati: Writer’s Digest Books, 1993.), and Georges Polti claims that there are thirty six (The Thirty-Six Dramatic Situations. trans. Lucille Ray). So the story goes.

All of these discussions about plot typically involve some discussion of a story’s central conflict. But discussions of conflict are more appropriately classified as fabula. Nevertheless, many of these same discussions also explore the flow, or trajectory, of the narrative, and these I consider to be appropriately categorized as syuzhet. Often these discussions of plot engage visualization in order to convey the “movement” of the narrative. Perhaps the best example of this is the one offered by Vonnegut.

A Significant Problem

Still, all of these explanations of plot suffer from a significant problem: a lack of data. Each of these proposed taxonomies suffers from anecdotalism. Vonnegut draws the plot of Cinderella for us on his chalk board, and we can imagine a handful of similar plot shapes. He describes another plot and names it “man in hole,” and we can imagine a few similar stories. But our imaginations are limited.

This limitation led me to think hard about the problem of how to compare, mathematically and computationally, the shape of one story to another. Assuming I could use computers and some NLP magic to extract plot shape from narrative (see A Novel Method for Detecting Plot), it would still be impossible to compare one shape to another because of the simple fact that stories are not the same length. Vonnegut solved this problem by creating an x-axis that runs from B to E, that is, from beginning to end. What Vonnegut did not solve, however, was the real computational problem of text length.

It was tempting to consider simply breaking each book into ten or one-hundred equally sized pieces and then taking measurements of the mean emotional valence in each chunk.

Unfortunately, some of the books would have much larger chunks and with larger chunks would come the possibility of more and more diverse valence markers. What happens, in fact, is that larger chunks of text tend to have a preponderance of both positive and negative valence markers. The end result is that all the means end up very close to neutral on the y-axis of emotional valence. Indeed, books as a whole tend to have a mean valence close to zero on a scale of -1 to 1. (I tested this by calculating the mean valence for 3500 novels in my nineteenth century novels corpus and then plotting the results as a histogram. The distribution showed a clustering around zero with very few books on the extremes.)

So, I needed a way to deal with length. I needed a way to compare the shapes of the stories regardless of the length of the novels. Luckily, since coming to UNL, I’ve become acquainted with a physicist who is one of the team of scientists who recently discovered the Higgs Boson at CERN. Over coffee one afternoon, this physicist, Aaron Dominguez, helped me figure out how to travel through narrative time.

A Solution

Aaron introduced me to a mathematical formula from signal processing called the Fourier transformation. The Fourier transformation provides a way of decomposing a time based signal and reconstituting it in the frequency domain. A complex signal (such as the one seen above in the first figure in this post) can be decomposed into series of symmetrical waves of varying frequencies. And one of the magical things about the Fourier equation is that these decomposed component sine waves can be added back together (summed) in order to reproduce the original wave form–this is called a backward or reverse transformation. Fourier provides a way of transforming the sentiment-based plot trajectories into an equivalent data form that is independent of the length of the trajectory from beginning to end. The frequency domain begins to solve the book length problem.

It turns out that not all of these sine waves in the frequency domain are created equal; some play a bigger role in the construction of the original signal. In signal processing, a low-pass filter can be used to remove the background “hiss” in an audio recording, and a similar approach can be used to filter out the extremes in the sentiment trajectories. When a low-pass filter is applied to the sentiment data, it’s possible to filter and thereby smooth out a great deal of the affectual noise.

The filtered data from the frequency domain can then be reconstituted back into the time domain using the reverse transformation. At the same time, the x-axis can be normalized and the foundation shape of the story revealed.

Above you can see the core shape of Joyce’s Portrait revealed using the “bing” method of the get_sentiment function in the syuzhet package. (Check the package documentation and vignette for details on the various options and methods.)

Once a book’s plot trajectory is converted into this normalized space, we no longer have the problem of comparing books of different lengths. Compare the foundation shape of Joyce’s Portrait (above) to Wilde’s Picture of Dorain Grey (below).

The models reflect the key narrative movements in both of these plots. Young Stephen reaches a low point during and just after the sermon on hell which occurs midway through the narrative. Dorian’s life takes a dark turn as the reality of the portrait becomes apparent. But the full power of these transformed plots does not sit simply in visualization. The values that inform these visualizations can now be compared. In a follow up post, I’ll discuss how I measured and compared 40,000+ plot shapes and then clustered the resulting data in order to reveal six common, perhaps archetypal, plot shapes. . .

A few weeks ago Ben Schmidt posted a provocative blog entry titled “Typical TV episodes: visualizing topics in screen time.” It’s worth a careful read. . .

Ben began by topic modeling the closed captioning data from a series of popular TV series and then visualizing the ten most common topics over the time span of each episode. In other words, the x-axis is time, and the y-axis is a measure of topical presence. The end result is something that begins to look a bit like what we could call plot.

Ben followed this post with an even more provocative one on 12/16/14 “Fundamental plot arcs, seen through multidimensional analysis of thousands of TV and movie scripts“. This post led a number of us (Underwood, Mimno, Cherny, etc.) to question what the approach might reveal if applied to novels . . .

In my own recent work, I have been attempting to model plot movement in narrative fiction by analyzing the rise and fall of emotional valence across narrative time. It has been clear to me, however, that my method is somewhat impoverished by a lack context for the emotions I am measuring; Ben’s topic-based approach to plot structure might be just the context I’m missing, and some correlation analysis might be just the right recipe . . . as usual, Ben has given us a lot to think about—i.e. Happy Holidays!

After following the discussion on Twitter and on Ben’s blog, David Mimno wrote to me about whipping up some of these topical plot lines based on the 500 Topic model that I had built for Macroanalysis. Needless to say, I thought this was a great idea. (David and I had previously revisited my topical data for an article in Poetics.) Within a few hours, David had run the entire collection of 500 topics and produced 500 graphs showing the general behavior of each topic across all of the 3,500 texts in my corpus. You will find the output of David’s work here: http://mimno.infosci.cornell.edu/novels/plot.html

In David’s short introductory paragraph, he calls our attention to two specific topic graphs, one for the topic labeled “school” and another labeled “punishment.” You can find my graphs for these two topics here (school) and here (punishment). In referencing these two plots, David calls our attention to one topic (school) that appears prominently at the beginnings of novels in this corpus (think Bildungsroman, perhaps?) and another topic (punishment) that tends to be prominent at the end of novels (think Newgate novels or Oliver Twist, perhaps?).

Like the data from Ben, this data David has mined from my 19th century novels topic model is incredibly rich and demands deeper inspection. I’ve only begun to digest it in bits, but I do observe that a lot of topics carrying negative valence seem to rise over the course of narrative time. This makes intuitive sense if we believe that the central conflict of a novel must grow more intense as the novel progresses. The exciting thing to do ext is to move from the macro to the micro scale and look at the individual novels within this larger context. Perhaps we’ll be able to identify archetypal patterns and then observe which novels stick to the archetypes and which digress. . . what a feast!

Luckily we have a whole new year to indulge!

I’m pleased to announce that Willard McCarty and I are leading a two-year set of summer institutes in digital humanities at the National Humanities Center. Here is the official announcement:

“The first of the National Humanities Center’s summer institutes in digital humanities, devoted to digital textual studies, will convene for two one-week sessions, first in June 2015 and again in 2016. The objective of the Institute in Digital Textual Studies is to develop participants’ technological and scholarly imaginations and to combine them into a powerful investigative instrument. Led by Willard McCarty (King’s College London and University of Western Sydney) and Matthew Jockers (University of Nebraska), the Institute aims to further the development of individual as well as collaborative projects in literary and textual studies. The Institute will take place in Chapel Hill, North Carolina, in 2015 and at the National Humanities Center in Research Triangle Park, North Carolina, in 2016.”

The first workshop will take place June 8 – 12. Applications are now open. See http://nationalhumanitiescenter.org/digital-humanities/application.html

Over the past few days, several people have written to ask what I thought about the article by Adam Kirsch in New Republic (“Technology Is Taking Over English Departments The false promise of the digital humanities.”) In short, I think it lacks insight and new knowledge. But, of course, that is precisely the complaint that Kirsch levels against the digital humanities. . .

Several months ago, I was interviewed for a story about topic modeling to appear in the web publication Nautilus. The journalist, Dana Mackenzie, wanted to dive into the “so what” question and ask how my quantitative and empirical methods were being received by literary scholars and other humanists. He asked the question bluntly because he’d read the Stanley Fish blog in the NYT and knew already that there was some push back from the more traditional among us. But honestly, this is not a question I spend much time thinking about, so I referred Dana to my UNL colleague Steve Ramsay and to Matthew Kirshenbaum at the University of Maryland. They have each addressed this question formally and are far more eloquent on the subject than I am.

What matters to me, and I think what should matter to most of us is the work itself, and I believe, perhaps naively, that the value of the work is, or should be, self-evident. The answer to the question of “so what?” should be obvious. Unfortunately, it is not always obvious, especially to readers like Kirsch who are not working in the sub fields of this massive big tent we have come to call “digital humanities” (and for the record, I do despise that term for its lack of specificity). Kirsch and others inevitably gravitate to the most easily accessible and generalized resources often avoiding or missing some of the best work in the field.

“So what?” is, of course, the more informal and less tactful way of asking what one sometimes hears (or might wish to hear) asked after an academic paper given at the Digital Humanities conference, e.g. “I was struck by your use of latent Dirichlet allocation, but where is the new knowledge gained from your analysis?”

But questions such as this are not specific to digital humanities (I was struck by your use of Derrida, but where is the new knowledge gained from your analysis). In a famous essay, Claude Levi-Strauss asked “so what” after reading Vladimir Propp’s Morphology of the Folktale. If I understand Levi-Strauss correctly the beef with Propp is that he never gets beyond the model; Propp fails to answer the “so what” question. To his credit, Levi-Strauss gives Propp props for revealing the formal model of the folktale when he writes that: “Before the epoch of formalism we were indeed unaware of what these tales had in common.”

But then, in the very next sentence, Levi-Strauss complains that Propp’s model fails to account for content and context, and so we are “deprived of any means of understanding how they differ.”

“The error of formalism” Levi-Strauss writes, is “the belief that grammar can be tackled at once and vocabulary later.” In short, the critique of Propp is just simply that Propp did not move beyond observation of what is and into interpretation of what that thing that is, means (Propp 1984).

To be fair, I think that Levi-Strauss gave Propp some credit and took Propp’s work as a foundation upon which to build more nuanced layers of meaning. Propp identified a finite set of 31 functions that could be identified across narratives; Levi-Strauss wished to say something about narratives within their cultural and historical context. . .

This is, I suppose, the difference between discovering DNA and making DNA useful. But bear in mind that the one ever depends upon the other. Leslie Pray writes about the history of DNA in a Nature article from 2008:

Many people believe that American biologist James Watson and English physicist Francis Crick discovered DNA in the 1950s. In reality, this is not the case. Rather, DNA was first identified in the late 1860s by a Swiss chemist. . . and other scientists . . . carried out . . . research . . . that revealed additional details about the DNA molecule . . . Without the scientific foundation provided by these pioneers, Watson and Crick may never have reached their groundbreaking conclusion of 1953.

(Pray 2008)

I suppose I take exception to the idea that the kind of work I am engaged in, because it is quantitative and methodological, because it seeks first to define what is, and only then to describe why that which is matters, must meet some additional criteria of relevance.

There is often a double standard at work here. The use of numbers (computers, quantification, etc.) in literary studies often triggers a knee jerk reaction. When the numbers come out, the gloves come off.

When discussing my work, I am sometimes asked whether the methods and approaches I advocate and employ succeed in bringing new knowledge to our study of literature. My answer is a firm and resounding “yes.” At the same time, I need to emphasize that computational work in the humanities can be simply about testing, rejecting, or reconfirming, what we think we already know. And I think that is a good thing!

During a lecture about macro-patters of literary style in the 19th century novel, I used the example of Moby Dick. I reported how in terms of style and theme Moby Dick is a statistical mutant among a corpus of 1000 other 19th century American novels. A colleague raised his hand and pointed out that literary scholars already know that Moby Dick is an aberration. Why bother computing a new answer to a question for which we already have an answer?

My colleague’s question says something about our scholarly traditions in the humanities. It is not the sort of question that one would ask a physicist after a lecture confirming the existence of the Higgs Boson! It is, at the same time, an ironic question; we humanists have tended to favor a notion that literary arguments are never closed!

In other words, do we really know that Moby Dick is an aberration? Could a skillful scholar/humanist/rhetorician argue the counter point? I think that the answer to the first question is “no” and the second is “yes.” Maybe Moby Dick is only an outlier in comparison to the other twenty or thirty American novels that we have traditionally studied along side Moby Dick?

My point in using Moby Dick was not to pretend that I had discovered something new about the position of the novel in the American literary tradition, but rather to bring new evidence and a new perspective to the matter and in this case fortify the existing hypothesis.

If quantitative evidence happens to confirm what we have come to believe using far more qualitative methods, I think that new evidence should be viewed as a good thing. If the latest Mars rover returns more evidence that the planet could have once supported life, that new evidence would be important and welcomed. True, it would not be as shocking or exciting as the first discovery of microbes on Mars, or the first discovery of ice on Mars, but it would be viewed as important evidence nevertheless, and it would add one more piece to a larger puzzle. Why should a discussion of Moby Dick’s place in literary history be any different?

In short computational approaches to literary study can provide complementary evidence, and I think that is a good thing.

Computational approaches can also provide contradictory evidence, evidence that challenges our traditional, impressionistic, or anecdotal theories.

In 1990 my dissertation adviser, Charles Fanning, published an excellent book titled The Irish Voice in America. It remains the definitive text in the field. In that book he argued for what he called a “lost generation” of Irish-American writers in the period from 1900 to 1930. His research suggested that Irish-American writing in this period declined, and so he formed a theory about this lost generation and argued that adverse social forces led Irish-Americans away from writing about the Irish experience.

In 2004, I presented new evidence about this period in Irish-American literary history. It was quantitative evidence showing not just why Fanning had observed what he had observed but also why his generalizations from those observations were problematic. Charlie was in the audience that day and after my lecture he came up to say hello. It was an awkward moment, but to my delight, Charlie smiled and said, “it was just an idea.” His social theory was his best guess given the evidence available in 1990, and he understood that.

My point is to say that in this case, computational and quantitative methods provided an opportunity for falsification. But just because such methods can provide contradiction or falsification, we must not get caught up in a numbers game where we only value the testable ideas. Some problems lend themselves to computational or quantitative testing; others do not, and I think that is a fine thing. There is a lot of room under the big tent we call the humanities.

And finally, these methods I find useful to employ can lead to genuinely new discoveries. Computational text analysis has a way of bringing into our field of view certain details and qualities of texts that we would miss with just the naked eye (as John Burrows and Julia Flanders have made clear). I like to think that the “Analysis” section of Macroanalysis offers a few such discoveries, but maybe Mr. Kirsch already knew all that? For a much simpler example, consider Patrick Juola’s recent discovery that J. K. Rowling was the author of The Cuckoo’s Calling, a book Rowling wrote under the pseudonym Robert Galbraith. I think Joula’s discovery is a very good thing, and it is not something that we already knew. I could cite a number of similar examples from research in stylometry, but this example happens to be accessible and appealing to a wide range of non-specialists: just the sort of simple folk I assume Kirsch is attempting to persuade in his polemic against the digital humanities.

Works Cited:

Propp, Vladimir. Theory and History of the Folktale. Trans. Ariadna Y. Martin and Richard Martin. Edited by Anatoly Liberman. University of Minnesota Press, 1984. 180

Pray, L. (2008) Discovery of DNA structure and function: Watson and Crick. Nature

I have just posted the syllabus for my spring macroanalysis class focusing on Characterization in Literature. The class is experimental in many senses of the word. We will be experimenting in the class and the class will be an experiment. If all goes according to plan, the only thing about this class that will be different from a research lab is the grade I have to assign at the end—that is the one remaining bit about collaborative learning that still kicks me . . .

To be successful everyone is going to have to be high-performing and self-motivated, me included. For me, at least, the motivation comes from what I think is a really tough nut to crack: algorithmic detection and analysis of character and character types. So far the work in this area has been largely about character networks: how is Hamlet related to Gertrude, etc. That’s good work, but it depends heavily upon the human coding of character metadata before processing. That is precisely why our early experiments at the Stanford Literary Lab focused on Drama. . . the character names are already explicit in the speaker markup. Beyond drama, there have been some important steps taken in the direction of auto-detection of character in fiction, such as those by Graham Sack and Elson et. al, but I think we still have a lot more stepping to do, a whole lot more.

The work I envision for the course will include leveraging obvious tools such as those for named entity recognition and then thinking through and dealing with the more complicated problems of pronoun disambiguation. But my deeper interest here goes far beyond simple detection of entities. The holy grail that I see here lies not in detecting the presence or absence of individual characters but in detecting and tracking character archetypes on a grand macroscale. What if we could begin to answer questions such as these:

• Are there different classes of villains in the 19th century novel?
• Do we see a rise in the number of minor characters over the 20th century?
• What are the qualities that define heroines?
• How, if at all, do those qualities change/evolve over time? (think Jane Austen’s Emma vs. Stieg Larsson’s Lisbeth).
• Etc.

We may get nowhere; we may fail miserably. (Of course if I did not already have a couple of pretty good ideas for how to get at these questions I would not be bothering. . . but that, for now, is the secret sauce 😉 )

At the more practical, “skills” level, I’m requiring students to learn and submit all their work using LaTeX! (This may prove to be controversial or crazy–I only learned LaTeX six months ago.) For that they will also be learning how to use the knitr package for R in order to embed R code directly into the LaTeX, and all of this work will take place inside the (awesome) R IDE, RStudio. Hold on to your hats; it’s going to be a wild ride!

[Below is the text of my introduction of Willard McCarty, winner of the 2013 Busa Award.]

As the chair of the awards committee that selected Prof. McCarty for this award it is my pleasure to offer a few words of introduction.

I’m going to go out on a limb this afternoon and assume that you already know that Willard McCarty is Professor of Humanities Computing and Director of the Doctoral Program in the Department of Digital Humanities at King’s College London, and that he is Professor in the Digital Humanities Research Group, University of Western Sydney and that he is a Fellow of the Royal Anthropological Institute (London). I’ll assume that you already know that he is Editor of the British journal, Interdisciplinary Science Reviews and that he’s founding Editor of the online seminar Humanist. And I am sure you know that Willard is recipient of the Canadian Award for Outstanding Achievement in Computing in the Arts and Humanities, and of the prestigious Richard W. Lyman Award of the National Humanities Center. You have probably already read his 2005 book titled Humanities Computing, and you know of his many, many other writings and musing.

So I’m not going to talk about any of that stuff.

And since I’m sure that everyone here knows that the Roberto Busa Award was established in 1998. I’m not going to explain how the Busa award was set up to recognize outstanding lifetime achievement in the application of information and communications technologies to humanities research.

No I’m not going to say anything about that either.

Instead, I wish to say a few words about this fellow here.

This is Obi-Wan McCarty. Long before I met him in person, he had become a virtual friend, model, and mentor.

I began computing in the humanities in 1993, and like so many of us in those early days I was a young maverick with little or no idea what had been done before. Those were the days before the rebellion, when the dark forces of the Empire were still quite strong. It was a time when an English major with a laptop was considered a dangerous rebel. At times I was scared, and I felt alone in a dark side of a galaxy far, far, away.

And then somewhere between 1993 and 2001 I began to sense a force in the galaxy.

One day, in early 2001, I was walking with my friend Glen Worthey, and I mentioned how I had recently discovered the Humanist list and how there had been this message posted by Willard McCarty with the cryptic subject line “14.”

“Ah yes,” Glen said, “Obi-Wan McCarty. The force is strong with him.”

Message 14 from Obi-Wan was a birthday message. Humanist was 14 that day and Willard began his message with a reflection on “repetition” and how frequently newcomers to the list would ask questions that had already been asked. Rather than chastise those newbies, and tell them to go STFA (search the freakin’ archive), Willard encouraged them. He wrote in that message of how “repetition is a means of maintaining group memory.” I was encouraged by those words and by Willard’s ongoing and relentless commitment not simply to deep, thoughtful, and challenging scholarship, but to nurturing, teaching, welcoming, and mentoring each new generation.

So Willard, thank you for your personal mentorship, thank you for continuing to demonstrate that scholarly excellence and generosity are kindred spirits. Congratulations on this award. May the force be with you.

Back in November, Julia Flanders and I were invited to stage a debate on the matter of “scale” in digital humanities research for the “Boston Area Days of DH” conference keynote: Julia was to represent the micro scale and I the macro.

Julia and I met up during the MLA conference in January and began sketching out how the talk might go. The first thing we discovered, of course, is that we did not in fact have a real difference of opinion on this matter of scale. Big data, small data, close reading and distant . . . these things matter much less than what a scholar actually decides to do and say. In other words, we were both ultimately interested in new knowledge and not too much concerned with the level of scale necessary to derive that new knowledge.

In other words, it’s a false and probably irrelevant debate. And while we agreed on this point in general terms, we discovered in the course of composing and editing the script for our mock debate that there were legitimate nuances that deserved to be put into the light of day. The script form our “debate” and all of the slides are now available via UNL’s open access repository as “A Matter of Scale.”

Julia has posted a few comments on the experience of co-authoring this presentation with me on her blog. Check it out at http://juliaflanders.wordpress.com/2013/03/28/a-matter-of-scale/.

[For the “Theories and Practices of the Literary Lab” roundtable at MLA yesterday, panelists were asked to speak for 5 minutes about their vision of a literary lab. Here are my remarks from that session–#147]

I take the descriptor “literary lab” literally, and to help explain my vision of a literary lab I want to describe how the Stanford Literary Lab that I founded with Franco Moretti came into being.

The Stanford Lab was born out of a class that I taught in the fall of 2009. In that course I assigned 1200 novels and challenged students to explore ways of reading, interpreting, and understanding literature at the macro-scale, as an aggregate system. Writing about the course and the lab that evolved from the course, Chronicle of Higher Ed reporter Marc Parry described it as being based on: “a controversial vision for changing a field still steeped in individual readers’ careful analyses of texts.” That may be how it looks from the outside, but there was no radical agenda then and no radical agenda today.

In the class, I asked the students to form into two research teams and to construct research projects around this corpus of 1200 novels. One group chose to investigate whether novel serialization in the 19th century had a detectable/measurable effect upon novelistic style. The other group pursued a project dealing with lexical change over the century, and they wrote a program called “the correlator” that was used to observe and measure semantic change.

After the class ended, two students, one from each group asked to continue their work as independent study; I agreed. Over the Christmas holiday, word spread to the other students from the seminar and by the New Year 13 of the original 14 in the seminar wanted to keep working. Instead of 13 independent studies, we formed an ad-hoc seminar group, and I found an empty office on the 4th floor where we began meeting, sometimes for several hours a day. We began calling this ugly, windowless room, the lab.

Several of the students in my fall class were also in a class with Franco Moretti and the crossover in terms of subject matter and methodology was fairly obvious. As the research deepened and became more nuanced, Franco began joining us for lab sessions and over the next few months other faculty and grad students were sucked into this evolving vortext. It was a very exciting time.

At some point, Franco and I (and perhaps a few of the students) began having conversations about formalizing this notion of a literary lab. I think at the time our motivation had more to do with the need to lobby for space and resources than anything else. As the projects grew and gained more steam, the room got smaller and smaller.

I mention all of this because I do not believe in the “if we build it they will come” notion of digital humanities labs. While it is true that they may come if we build them; it is also true, and I have seen this first hand, that they may come with absolutely no idea of what to do.

First and foremost a lab needs a real and specific research agenda. “Enabling Digital Humanities projects” is not a research agenda for a lab. Advancing or enabling digital humanities oriented research is an appropriate mission for a Center, such as our Center for Digital Humanities Research at Nebraska, but it is not the function of a lab, at least not in the limited literal sense that I imagine it. For me, a lab is not specifically an idea generator; a lab is a place in which ideas move from birth to maturation.

It would be incredible hyperbole to say that we formally articulated any of this in advance. Our lab was the opposite of premeditated. We did, however, have a loosely expressed set of core principles. We agreed that:

1. Our work would be narrowly focused on literary research of a quantitative nature.
2. All research would be collaborative, even when the outcome ends up having a single author.
3. All research would take the form of “experiments,” and we would be open to the possibilities of failure; indeed, we would see failure as new knowledge.
4. The lab would be open to students and faculty at all levels–and, on a more ad hoc basis, to students and faculty from other institutions.
5. In internal and external presentation and publication, we would favor the narrative genre of “lab reports” and attempt to show not only where we arrived, but how we got there.

I continue to believe that these were and are the right principles for a lab even while they conflict with much about the way Universities are organized.

In our lab we discovered that to focus, to really focus on the work, we had to resist and even reject some of the established standards of pedagogy, of academic hierarchy, and of publishing convention. We discovered that we needed to remove instructional barriers both internal and external in order to find and attract the right people and the right expertise. We did not do any of this in order to make a statement. We were not academic radicals bent on defying the establishment.

Nor should I leave you with the impression that we figured anything out. The lab remains an organic entity unified by what some might characterize as a monomaniacal focus on literary research. If there was any genius to what we did, it was in the decision to never compromise our focus, to do whatever was necessary to keep our focus on the literature.

In preparation for a panel session at DH Commons today, I was asked to consider the question: “What one step would you recommend a newcomer to DH take in order to join current conversations in the field?” and then speak for 3 – 4 minutes. Below is the 5 minute version of my answer. . .

With all the folks assembled here today, I figured we’d get some pretty good advice about what constitutes DH and how to get started, so I decided that I ought to say something different from what I’d expect others to say. I have two specific bits of advice, and I suppose that the second bit will be a little more controversial.

But let me foreground that by going back to 2011 when my colleague Glen Worthey and I organized the annual Digital Humanities conference at Stanford around a big tent, summer of love theme. We flung open the flaps on the Big Tent and said come on in . . . We believed, and we continue to believe, that there is a wide range of very good and very interesting work being done in “digital humanities.” We felt that we needed a big tent to enclose all that good work. But let’s face it, inside the big tent it’s a freakin’ three ring circus. Some folks like clowns and others want to see the jugglers. The DH conference is not like a conference on Victorian Literature. And that, of course, is the charm and the curse.

While it probably makes sense for a newcomer to poke around and gain some sense of the “disciplinary” history of the “field.” I think the best advice I can give a newcomer is to spend very little time thinking about what DH is and spend as much time as possible doing DH.

It doesn’t really matter if the world looks at your research and says of it: “Ahhhh, that’s some good Digital Humanities, man.” What matters, of course, is if the world looks at it and says, “Holy cow, I never thought of Jane Austen in those terms” or “Wow, this is really strong evidence that the development of Roman road networks was entirely dependent upon seasonal shifts.” The bottom line is that it is the work you do that is important, not how it gets defined.

So I suppose that is a bit of advice for newcomers, but let me answer the question more concretely and more controversially by speaking as someone who hangs out in one particular ring of the DH Big Tent.

If you understand what I have said thus far, then you know that it is impossible to speak for the Digital Humanities as a group, so, for some, what I am going to say is going to sound controversial. And if I hear that one of you newcomers ran out at the end of this session yelling “Jockers thinks I need to learn a programming language to be a digital humanist,” then I’m going to have to kick your butt right out of the big tent!

Learning a programming language, though, is precisely what I am going to recommend. I’m even going to go a bit further and suggest a specific language called R.

By recommending that you learn R, I am also advocating learning some statistics. R is primarily a language used for statistical computing, which is more or less the flavor of Digital Humanities that I practice. If you want to be able to read and understand the work that we do in this particular ring of the big tent you will need some understanding of statistics; if you want to be able to replicate and expand upon this kind of work, you are going to need to know a programming language, so I recommend learning some R and killing two birds with one stone.

And for those of you who don’t get turned on by p-values, for loops, and latent dirichlet allocation, I think learning a programing language is still in your best interests. Even if you never write a single line of code, knowing a programming language will allow you to talk to the natives, that is, you will be able to converse with the non-humanities programmers and web masters and DBAs and systems administrators, who we so often collaborate with as digital humanists. Whether or not you program yourself, you will need to translate your humanistic questions into terms that a non-specialist in the humanities will understand. You may never write poetry in Italian, but if you are going to travel in Rome, you should at least know how to ask for directions to the coliseum.

I could not make it to the DH conference in Hamburg this year (though I did manage to appear virtually). As chair of the Busa Award committee I had the pleasure of announcing that Willard McCarty had won the award. Willard will accept the award in 2013 when DH meets at the University of Nebraska. Here is the text of my announcement which was read today in Hamburg:

I was very pleased to serve as the Chair of the Busa Award committee this cycle, and though I am disappointed that I was unable to travel to Hamburg this year to make this announcement in person, I’m delighted with the end result. I am also delighted that the award will be given at the 2013 conference hosted by the University of Nebraska. Having recently joined the faculty there, I’m quite certain I will be attending next year’s meeting!

The winner of the 2013 Busa Award is a man of legendary kindness and generosity. His contributions to the growth and prominence of Digital Humanities will be familiar to us all. He is a gentleman, a scholar, a philosopher, and a long time fighter for the cause. He is, by one colleague’s accounting, the “Obi-Wan Kenobi” of Digital Humanities. And I must concur that “the force” is strong with this one. Please join me in congratulating Willard McCarty on his selection for the 2013 Busa Award.

In the last chapter of forthcoming my book, I write about the challenges of copyright law and how many a digital humanist is destined to become a 19th-centuryist if the law isn’t reformed to specifically allow for and recognize the importance of “non-expressive” use of digitized content.*

This week the Amicus Brief that I co-authored with Matthew Sag and Jason Schultz was submitted. The brief (see Brief of Digital Humanities and Law Scholars as Amici Curiae in Authors Guild, Inc. Et Al V. Hathitrust Et Al.) includes official endorsement from the Association of Computers in the Humanities as well as the support and signature of many individual scholars working in the field.

* “Non-expressive use” is Matthew Sag’s far more pleasing formulation of what many have come to call “non-consumptive use.”

Earlier this week Kathryn Schultz of the New York Times published a rather provocative, challenging, and in my opinion under-researched and over-sensationalized article about my colleague Franco Morreti’s work theorizing a mode of literary analysis that he has termed “distant-reading.” Others have already pointed out some of the errors Schultz made, and I’m fairly certain Moretti would be happy to clarify any confusion Schultz may have about his work if she were to actually interview him (i.e. before paraphrasing him). My interest here is to offer some specific thoughts and some background on “distant-reading” or what I have preferred to call “macroanalysis.”[1]

The approach to the study of literature that I call macroanalysis, instead of distant-reading (for reasons explained below), is in general ways akin to the social-science of economics or, more specifically, macroeconomics. Before the 20th century there wasn’t a defined field of “Macroeconomics.” There was, however, microeconomics, which studies the economic behavior of individual consumers and individual businesses. As such, microeconomics can be seen as analogous to the study of individual texts via “close-readings” of the material. Macroeconomics, however, is about the study of the entire economy. It tends toward enumeration and quantification and is in this sense similar to literary inquiries that are not highly theorized: bibliographic studies, biographical studies, literary history, philology, and the enumerative analysis that is the foundation of humanities computing.

By way of an analogy, we might think about interpretive close-readings as corresponding to microeconomics while quantitative macroanalysis corresponds to macroeconomics. Consider, then, that in many ways the study of literary genres or literary periods is a type of macro approach to literature. Say, for example, a scholar specializes in early 20th century poetry. Presumably, this scholar could be called upon to provide sound generalizations, or “distant-readings” about 20th century poetry based on a broad reading of individual works within that period. This would be a sort of “macro-, or distant-, reading” of the period. But this parallel falls short of approximating for literature what macroeconomics is to economics, and it is in this context that I prefer the term macroanalysis over distant-reading. The former term places the emphasis on the quantifiable methodology over the more interpretive practice of “reading.” Broad attempts to generalize about a period or about a genre are frequently just another sort of micro-analysis, in which multiple “cases” or “close-readings” of individual texts are digested before generalizations about them are drawn in very qualitative ways. Macroeconomics, on the other hand, is a more number-based discipline, one grounded in quantitative analysis not qualitative assessments. Moreover, macroeconomics employs a number of quantitative benchmarks for assessing, scrutinizing, and even forecasting the macro-economy. While there is an inherent need for understanding the economy at the micro level, in order to contextualize the macro-results, macroeconomics does not directly involve itself in the specific cases, choosing instead to see the cases in the aggregate, looking to those elements of the specific cases that can be generalized, aggregated, and quantified.

Micro-oriented approaches to literature, highly interpretive readings of literature, remain fundamentally important. Just as microeconomics offers important perspectives on the economy. It is the exact interplay between the macro and micro scale that promises a new, enhanced, and perhaps even better understanding of the literary record. The two approaches work in tandem and inform each other. Human interpretation of the “data,” whether it be mined at the macro or micro level, remains essential. While the methods of enquiry, of evidence gathering, are different, they are not antithetical, and they share the same ultimate goal of informing our understanding of the literary record, be it writ large or small. The most fundamental and important difference in the two approaches is that the macroanalytic approach reveals details about texts that are for all intents and purposes unavailable to close-readers of the texts. Writing of John Burrows’s study of Jane Austen’s oeuvre, Julia Flanders points out how Burrows’s computational study brings the most common words such as “the” and “of” into our field of view.

Flanders writes: “His [Burrows] effort, in other words, is to prove the stylistic and semantic significance of these words, to restore them to our field of view. Their absence from our field of view, their non-existence as facts for us, is precisely because they are so much there, so ubiquitous that they seem to make no difference.” (Flanders 2005)

At its most basic, the macroanalytic approach I’m advocating is simply another method of gathering information about texts, of accessing the details. The information is different from what is derived via close reading, but it not of lesser or greater value to scholars for being such.

Flanders goes on: “Burrows’ approach, although it wears its statistics prominently, foreshadows a subtle shift in the way the computer’s role vis-á-vis the detail is imagined. It foregrounds the computer not as a factual substantiator whose observations are different in kind from our own—because more trustworthy and objective—but as a device that extends the range of our perceptions to phenomena too minutely disseminated for out ordinary reading.” (Flanders 2005)

A macroanalytic approach not only helps us to see and understand the larger “literary economy” but, by means of its scope, to better see and understand the degree to which literature and the individual authors who manufacture the literature respond to or react against literary and cultural trends within their realm of experience. If authors are inevitably influenced by their predecessors, then we may even be able to chart and understand “anxieties of influence” in concrete, quantitative ways.

For historical and stylistic questions in particular, the macroanalytic approach has distinct advantages over the more traditional practice of studying literary periods and genres by means of a close study of “representative” texts. Speaking of his own efforts to provide a more encompassing view of literary history, Franco Moretti writes that “a field this large cannot be understood by stitching together separate bits of knowledge about individual cases, because it isn’t a sum of individual cases: it’s a collective system, that should be grasped as a whole . . .” (2005). To generalize about a “period” of literature based on a study of a relatively small number of books is to take a significant leap. It is less problematic, though, to consider how a macroanalytic study of several thousand texts might lead us to a better understanding of the individual texts. Until recently, we have not had the opportunity to even consider this later option, and it seems reasonable to imagine that we might, through the application of both approaches, reach a new and better informed understanding of our primary materials. This is what Juri Tynjanov imagined in 1927: “Strictly speaking”, writes Tynjanov, “one cannot study literary phenomena outside of their interrelationships.” Fortunately for me and for scholars such as Moretti, the multitude of interrelationships that overwhelmed and eluded Tynjanov and pushed the limits of close-reading can now be explored with the aid of computation, statistics and huge digital libraries.

My book on this subject, Literary Studies, the Digital Library, and the Inevitability of Influence, is now under contact with [Update: will be published in 2013 as Macroanalysis: Digital Methods and Literary History by University of Illinois Press.

[1] I began using the term macroanalysis in late 2003. At the time, Moretti and I were putting together plans for a co-taught course titled “Electronic Data and Literary Theory.” The course we imagined would be a research seminar in the full sense of the word and in our syllabus (dated 11/3/2003) we wrote: “the main purpose of this seminar is methodological rather than historical: learning how to use electronic search systems to analyze large quantities of data — and hence get a new, better understanding of literary and cultural history.” During the course I began work developing a text analysis toolkit that I later called CATools (for Corpus Analysis Tools). In terms of methodology, I was learning a lot at the time from work in corpus linguistics but also discovering that we (literary folks) have an entirely different set of questions. So it made sense to do at least a bit of wheel reinvention. My first experiments with the macroanalytic methodology were constructed around a corpus of Irish-American novels that I had been building since my dissertation research. I presented the first results of this work in Liverpool, at the 2004 meeting of the American Conference for Irish Studies. My paper, titled “Making and Mining a Digital Archive: the Case of the Irish-American West Project,” was part how-to and part results–I’d made one non-trivial discovery about Irish-American literary history based on this new methodology. In the spring of 2005, I offered a more detailed methodological overview of the toolkit at the inaugural meeting of the Text Analysis Developer’s Alliance. An overview of my project was documented on the TADA blog. Later that summer (2005), I presented a more generalized methodological paper titled “A Macro-Economic Model for Literary Research” at the joint meeting of the ACH and ALLC in Victoria, BC. It was there that I first articulated the economic analogy that I have come to find most useful for explaining Moretti’s idea of “distant-reading.” In 2006, while I was in residence as Research Scholar in the Digital Humanities at the Stanford Humanities Center, I spent a good deal of time thinking about macro-scale approaches to literature and then writing corpus analysis code . By the summer of 2007, I had developed a whole new toolkit and presented the first significant findings in a paper titled “Macro-Analysis (2.0)” which I delivered at the 2007 Digital Humanities meeting in Illinois. Coincidentally, this was the same conference at which Moretti presented the opening keynote lecture, a paper exploring a corpus of 19th century novel titles, which would eventually be published in Critical Inquiry. That research utilized software that I had developed in the CATools package.

Rowfont Press of Wichita, Kansas has just published a newly illustrated edition of Charles Driscoll’s memoir Kansas Irish (with my Critical Introduction). The book is available at Amazon. Kansas Irish and the two sequels that follow provide the most complete and authentic rendering of Irish life on the American prairie in the 19th Century.

Several months ago, a group of us from the Stanford Literary Lab wrote and sent out for review the article that now appears in Pamphlet 1 of the Lab. The article, titled “Quantitative Formalism: an Experiment” was submitted, peer-reviewed, and approved for publication in a prestigious literary journal. There was, however, a catch. The editors of the journal asked that we trim the number of charts in the article and that we alter the tone and character of the article to make it less of a narrative. In other words, the article was of a style and content that the editors found to be too foreign to their traditions.

Rather than revise the article in ways we felt would misrepresent its function and intent, we turned to that most traditional of literary forms, the pamphlet. Considering the largely quantitative and digital methodology employed in our research, a pamphlet was a seemingly ironic choice. The most obvious venue would seem to have been the web. After all, aren’t the blog and the web site, the pamphlet forms of the digital age (see, for example, Pamphleteers and Web Sites)? We certainly considered an electronic format, and we have posted a pdf version of the essay on our web site, but we decided to print.

Why print the pamphlet? As a literary form, the pamphlet has a long tradition of going against the grain; it’s an alternative form that is malleable. In the pamphlet, George Orwell wrote, “one has complete freedom of expression, including, if one chooses, the freedom to be scurrilous, abusive, and seditious; or, on the other hand, to be more detailed, serious and ‘high-brow’ than is ever possible in a newspaper or in most kinds of periodicals.” It has been used in campaigning and marketing, but most famously, the pamphlet has been employed by political, religious, and social “provocateurs” and “radicals.” Many of these objects have been lost, but the best have been preserved, such as those crafted by talented satirists (Swift) sober thinkers (Paine) and social critics (Voltaire). And they are preserved because they are historical objects, “ephemera” as the librarians say. The pamphlet is an ephemeral object, an object “lasting only for a day.” As “an experiment” our “quantitative formalism” pamphlet is a middle point, a hash mark on the line of time, not an end point or destination not even a beginning.

It’s interesting to consider how the preferred citation style of literary scholarship, the MLA Style, places emphasis on page reference over time, over moment of publication. With some obvious exceptions, the basic logic here is that what someone says about Shakespeare today has the same validity, the same scholarly purchase, as something said fifty years ago. As a discipline, our “results” tend to be qualitative and interpretive, not bound in time or subject to revision based on the introduction of new evidence. They are, of course, subject to reinterpretation, but that is fundamentally different from the changes wrought by new discoveries. The date-based citation style, on the other hand, places emphasis on points in time and acknowledges the fast-paced and ephemeral nature of certain fields of research. This is most obvious in the sciences, in medicine, for example, where new experiments and new discoveries are constantly and quickly changing the field. One need only eavesdrop on the the Digital Humanities twittersphere for a few hours to note the similarities; the pace of change is rapid.

So why not a pamphlet? Why not recognize in both form, title, and narrative style that ours is an experiment? It is a bit of research that is useful for today but also something we entirely expect to change. Indeed, we are already working on the next iteration(s), the next experiments. Unlike the neatly closed arguments of our traditional work, these experiments of our Literary Lab open as many doors as they close. In fact, in the course of our research on novel genres, it became apparent that we could, must, go on forever. Each test led us to some new idea, some new direction to explore. There are some discoveries to be sure, and some of our results will likely, hopefully, stand the test of time. But my co-authors and I understand, or perhaps simply “believe,” that there is still much, much more work to be done. If it reads more like a lab report than a traditional essay, it’s because it is a lab report and self-consciously so, intentionally so.

Readers wishing to experience the full pleasure of a touchable paper pamphlet may contact me with their name and address. No charge, while supplies last:-)

I’ve been hearing a lot about “collaboration,” especially in the digital humanities. Lisa Spiro at Rice University has written a very informative post about Collaborative Authorship in the Humanities as well as another post providing Examples of Collaborative Digital Humanities Projects. Both of these posts are worth reading, and Spiro offers some well-thought out and researched perspectives.

My own experiences with collaboration include both research and authorship. I have seen first hand how fruitful collaboration, especially interdisciplinary collaboration, can be. It is safe to say that I’m a believer. In fact the course I have been teaching for the last two years, Literary Studies and the Digital Library, is designed entirely around collaborative research projects. And yet I have to say that I’m am entirely suspicious of the current rage for “collaboration.”

No doubt the current popularity of collaboration, at least in the humanities, is a natural extension of the movement toward interdisciplinary studies. Through collaboration with people outside our individual disciplines has led to fruitful work, there seems to be an unnatural desire on the part of some administrators and even some colleagues to “foster collaboration,” as if collaboration were something that occurs in a petri dish, something that needs only to be “fostered” in order to evolve.

But collaboration does not arise out of a petri dish, it arises out of need. Sure, there are serendipitous collaborations that arise out of proximity: X bumps into Y at the water cooler and they get to talking . . . but more often successful collaboration arises out of need: X wishes to investigate a topic but requires the skills of Y in order to do a good job.

Failed collaborations, on the other hand, are all too often the result of good intentioned but overly forced attempts to bring people together. I attended a seminar a couple of years ago on the subject of “fostering collaboration in the humanities.” The organizers of the meeting certainly understood the promise of new knowledge that might be derived through interaction, but they entirely miscalculated when it came to individual motivation to collaborate. It’s a classic case of putting the cart before the horse. In my experience, fruitful collaboration evolves organically and is motivated by the underlying research questions, questions that are always too big and too complex to be addressed by a single researcher.

Those who do corpus level computational text analysis are always hungry for more and more texts to analyze. Though we’ve become adept at locating texts from a wide range of sources (our own institutional repositories as well as a number of other places including Google Books, the Internet Archive, and Project Gutenberg), we still face a number of preprocessing tasks to bring those various files into some standard format. The texts found at these resources are not always in a format friendly to the tools we use for processing those texts. For example, I’ve developed lots of processing scripts that are designed to leverage the metadata that is frequently encoded into TEI-based xml. A text from Project Gutenberg, however, is not only just plain text, but it has a lot of boilerplate text at the beginning and end of each file that needs to be removed prior to text analysis.

I’m currently building a corpus of 19th century novels and discovered that many of the texts I would like to include have already been digitized by Project Gutenberg. This, of course, was great news. But, the system I have developed for ingesting texts into my corpus assumes that the texts will all be in TEI-XML with markup indicating such important things as “author,” “title”, and “date” of publication. I downloaded about 100 novels and was about to begin opening them up one by one and adding the metadata. . .eek! I quickly realized the mundanity of the task and thought, “hmm, I bet someone has written a nice regex script for doing this sort of thing.” A quick trolling of the web led me to the web page of Michiel Overtoom who had developed some python scripts for downloading and cleaning up (“beautifying” in his language) Dutch Gutenberg texts for his eBook Reader. Overtoom’s process is mainly designed to strip out the boilerplate and then rename the files with naming conventions that reflect the author and title of the books.

With Overtoom’s script as a base, I reengineered the code to convert a Gutenberg text into a minimally encoded and TEI-compliant XML file. The script builds a teiHeader that includes the author and title of the work (unfortunately, Project Gutenberg texts do not include publication dates, why?) and then adds “text”, “body”, div, and all the p tags. The final result is a document that meets basic TEI requirements. The script is copied below, but since the all important python spacing may be destroyed by this posting, it’s better to download it here and then change the file extension from .txt. to “.py”. Enjoy!

# gutenbergToTei.py
#
# Reformats and renames etexts downloaded from Project Gutenberg.
#
# Software adapted from Michiel Overtoom, motoom@xs4all.nl, july 2009.
# 
# Modified by Matthew Jockers August 17, 2010 to encode result into TEI based XML
#
# October 26, 2015: Peeter Tinits notes that for non-latin characters ', encoding="utf8"' could be added to both "open" functions.

import os
import re
import shutil

remove = ["Produced by","End of the Project Gutenberg","End of Project Gutenberg"]

def beautify(fn, outputDir, filename):
    ''' Reads a raw Project Gutenberg etext, reformat paragraphs,
    and removes fluff.  Determines the title of the book and uses it
    as a filename to write the resulting output text. '''
    lines = [line.strip() for line in open(fn)]
    collect = False
    lookforsubtitle = False
    outlines = []
    startseen = endseen = False
    title=""
    one="<?xml version=\"1.0\" encoding=\"utf-8\"?><TEI xmlns=\"http://www.tei-c.org/ns/1.0\" version=\"5.0\"><teiHeader><fileDesc><titleStmt>"
    two = "</titleStmt><publicationStmt><publisher></publisher><pubPlace></pubPlace><availability status=\"free\"><p>Project Gutenberg</p></availability></publicationStmt><seriesStmt><title>Project Gutenberg Full-Text Database</title></seriesStmt><sourceDesc default=\"false\"><biblFull default=\"false\"><titleStmt>"
    three = "</titleStmt><extent></extent><publicationStmt><publisher></publisher><pubPlace></pubPlace><date></date></publicationStmt></biblFull></sourceDesc></fileDesc><encodingDesc><editorialDecl default=\"false\"><p>Preliminaries omitted.</p></editorialDecl></encodingDesc></teiHeader><text><body><div>"
    for line in lines:
        if line.startswith("Author: "):
        	author = line[8:]
        	authorTemp = line[8:]
        	continue
        if line.startswith("Title: "):
            title = line[7:]
            titleTemp = line[7:]
            lookforsubtitle = True
            continue
        if lookforsubtitle:
            if not line.strip():
                lookforsubtitle = False
            else:
                subtitle = line.strip()
                subtitle = subtitle.strip(".")
                title += ", " + subtitle
        if ("*** START" in line) or ("***START" in line):
            collect = startseen = True
            paragraph = ""
            continue
        if ("*** END" in line) or ("***END" in line):
            endseen = True
            break
        if not collect:
            continue
        if (titleTemp) and (authorTemp):
        	outlines.append(one)
        	outlines.append("<title>")
        	outlines.append(titleTemp)
        	outlines.append("</title>")
        	outlines.append("<author>")
        	outlines.append(authorTemp)
        	outlines.append("</author>")
        	outlines.append(two)
        	outlines.append("<title>")
        	outlines.append(titleTemp)
        	outlines.append("</title>")
        	outlines.append("<author>")
        	outlines.append(authorTemp)
        	outlines.append("</author>")
        	outlines.append(three)
        	authorTemp = False
        	titleTemp = False
        	continue
        if not line:
            paragraph = paragraph.strip()
            for term in remove:
                if paragraph.startswith(term):
                    paragraph = ""
            if paragraph:
            	paragraph = paragraph.replace("&", "&")
                outlines.append(paragraph)
                outlines.append("</p>")
            paragraph = "<p>"
        else:
            paragraph += " " + line
			
    # Compose a filename.  Replace some illegal file name characters with alternatives.
    #ofn = author + title[:150] + ".xml"
    ofn = filename
    ofn = ofn.replace("&", "")
    ofn = ofn.replace("/", "")
    ofn = ofn.replace("\"", "")
    ofn = ofn.replace(":", "")
    ofn = ofn.replace(",,", "")
    ofn = ofn.replace(" ", "")
    ofn = ofn.replace("txt", "xml")
        
    outlines.append("</div></body></text></TEI>")
    text = "\n".join(outlines)
    text = re.sub("End of the Project Gutenberg .*", "", text, re.M)
    text = re.sub("Produced by .*", "", text, re.M)
    text = re.sub("<p>\s+<\/p>", "", text)
    text = re.sub("\s+", " ", text)
    f = open(outputDir+ofn, "wt")
    f.write(text)
    f.close()

sourcepattern = re.compile(".*\.txt$")
sourceDir = "/Path/to/your/ProjectGutenberg/files/"
outputDir = "/Path/to/your/ProjectGutenberg/TEI/Output/files/"

for fn in os.listdir(sourceDir):
    if sourcepattern.match(fn):
        beautify(sourceDir+fn, outputDir, fn)

Yesterday’s Chronicle of Higher Education ran an article by Marc Parry about the work we are doing here in our new Literature Lab with “big data.” It’s awfully nice to be compared to Lewis and Clark exploring the frontiers of literary scholarship, but I think the article fails to give due credit to the exceptional group of students who have been working with Franco Moretti and me in the Lab. Far from being the lab “grunts” that Parry calls them, these students are the lifeblood of the lab, and the projects we are working on spring from their ideas and their passion for literature.

I understand how such a confusion of roles could happen: in the science lab, students often work *under* a faculty member who has received a grant to pursue a particular line of research. Indeed, the funding of grad students in the sciences is often based on this model; the grant pays for the work they do in the lab.

Our literature lab is nothing at all like this; it is a far more egalitarian enterprise and there are no monetary incentives for the students. Instead, the motivation for the students is pure research, the opportunity to push the envelope and experience the excitement of discovering new territory. Yes, Moretti and I serve as guides, advisors, and mentors in this process, but it is important to emphasize the truly collaboration nature of the enterprise.

Moretti and I do not have the answers nor do we necessarily make up the questions. In the case of the most recent work, in fact, all the questions have come directly from the students themselves. A recent article by Amanda Chang and Corrie Goldman (“Stanford Students Use Digital Tools to Analyze Classic Texts“) captures the student’s role quite accurately. Our lab is based on the idea that any good question deserves to be pursued. Students or faculty may pose those questions and teams evolve organically based on interest in the questions. The result, of course, is that we *all* learn a lot.

As a teenager, one of my favorite films was The Magnificent Seven , a remake of the Seven Samurai with gunslingers played by Yul Brynner, Steve McQueen and other marquis names. The basic idea is that these gunslingers hire out to protect a small town being ravaged by bad guys. The excitement of the film comes as Brynner assembles his team of seven gunslingers, each with a special talent. They then train the local residents how to defend themselves and as they do the villagers and the gunmen develop a deep sense of respect and admiration for each other.

Working with this group of students in the lab has been a similar experience (without the guns). The work quite literally could not be done without the team and each member of the team has brought a unique talent to the project. One student in the group is an accomplished coder, another has read most every key book in the corpus, another has a penchant for math, another loves research. They are the magnificent seven, and I have never had the pleasure of working with a more talented group: yes, they are students but for me they are already colleagues. I trust their judgements and have a profound respect, and sometimes awe, for what they already know.

Parry’s article contains bits and pieces of an interview he conducted with Yale Professor Katie Trumpener. Speaking of our work and of Moretti’s notion of “distant-reading” Trumpener apparently said the following:

But what happens when his “dullard” descendants take up “distant reading” for their research?

“If the whole field did that, that would be a disaster,” she says, one that could yield a slew of insignificant numbers with “jumped-up claims about what they mean.”

“Dullard”? Really? I do hope that Ms. Trumpener’s comment was somehow taken out of context here and that she will very quickly write to the Chronicle to set the record straight. Otherwise I fear that some less forgiving souls might conclude that Ms. Trumpener is one herself. . .

UPDATE: Over the weekend I received a clarification via email. Ms. Trumpner writes: “I was referring to Moretti’s potential methodological “descendants”–ie. those coming after him, even long after him, not his current team-mates.” Ms. Trumpner notes that when she was interviewed the discussion was not about our Literature Lab at Stanford, but about Moretti’s approaches and general matters of how she and Moretti approach the study of literature. Her comments were made in that context and not in the context of a discussion of the current work of the lab.

Students in our new Literature Lab doing what English Majors do!

Folks keep expressing concern about the future of the humanities, and the “need” for a next big thing. In fact, the title of a blog entry in the April 23, 2010 New York Times takes it for granted that the humanities need “saving.” The blog entry is a follow up to an article from March 31, which explores how some literary critics are applying scientific methodologies and approaches to the study of literature. Of course, this isn’t really new. One only needs to read a few back issues of Literary and Linguistic Computing to know that we’ve been doing this kind of work for a long time (and even longer if one wants to consider the approaches suggested by the Russian Formalists). What is new is that the mainstream humanities and the mainstream press are taking notice.

In her response to the article, Blakey Vermeule (full disclosure, her office is just three doors from mine) makes a key point to take away from the discussion. She writes: “The theory wars are long gone and nobody regrets their passing. What has replaced them is just what was there all along: research and scholarship, but with a new openness to scientific ideas and methods” (emphasis mine). Before explaining why this is the key take-away, a little story. . .

Not too long ago, a colleague took me aside and asked in all earnestness, “what do I need to do to break into this field of Digital Humanities.” This struck me as a rather odd question to ask: a very clear putting of the cart before the horse. Digital Humanities (DH) is a wide stretching umbrella term that attempts to encompass and describe everything from new media theory and gaming to computational text analysis and digital archiving. There is a lot of room under the umbrella, and it really is, therefore, impossible to think of DH as a unified “field” that one can break into. In fact, there are no barriers to entry at all; the doors are wide open, come on in.

But, I’ll go out on a limb and argue that the DH community can be split into two primary groups: Group “A” is composed of researchers who study digital objects; Group “B” is composed of researchers who utilize digital tools to study objects (digital or otherwise). Group A, I would argue, is primarily concerned with theoretical matters and group B with methodological matters. In reality, of course, the lines are blurry, but this is a workable clarification.

. . . What Vermuele and others are describing in the New York Times business falls most cleanly into Group B. But I would not describe this movement toward empirical methodologies as revolutionary: when interested in certain types of questions, an empirical methodology just makes good common sense. I came to utilize computation in my research not because the siren’s song of revolution was tempting me away from my dusty, tired, and antiquated approaches to literature. Rather, computational tools and statistical methods simply offered a way of asking and exploring the questions that I (and others such as those pictured above) have about the literary field. What has changed is not the object of study but the nature of the questions.

So, the answer to my colleague who asked what is needed to “break into this field of Digital Humanities” is simply this: questions, you need questions.

At the recent Digital Humanities Conference in Maryland, Matthew Wilkins and I got into a discussion about famous authors and the “industries” of scholarship that their works have inspired (see Matt’s blog post about our discussion and his survey analysis of the MLA bibliography).

The first time I ever heard the term “industry” used in this context was in reference to the scholarship generated by Joyce’s novel Ulysses. As Joyce himself predicted (bragged) the book would keep scholars busy for centuries to come, and, of course, Joyce was right–well maybe not centuries, but you get the idea. But can we really compare the Shakespeare “industry” to the Joyce “industry” given that the Bard had such a significant head start in terms of establishing his scholarly “fan base”?

Using the MLA bibliography, Matthew W. took a stab at this and compiled some rough figures of recent scholarship on the two masters. By Matt’s count, since 1923, Joyce has inspired just 9315 citations to Shakespeare’s massive 35,489.

But there is an obvious problem here: the figures begin in 1923 and Ulysses, the book that really puts Joyce on the map, was only published in 1922. So Joyce is getting into the industry-building business a bit late. Clearly we must do some norming here to account for the Bard’s head start.

Now, since I am pretty sure that I owe Matt a beer if the Bard has a bigger industry, I think some well thought out math is warranted here:-) . . .

Shakespeare dies in 1616 and Joyce dies in 1941. Subtracting each death date from the last year of Matt’s analysis (2008) means that Shakespeare had 392 years to develop his industry and Joyce only 67 years. If we divide the total number of citations Matt found in the MLA bibliography by the total number of industry-building years, then the figures tell a very different story. Joyce averages 139 citations per year whereas Shakespeare manages only a paltry 90.5

But wait, there’s more. . . Querying the MLA bibliography using the search terms “shakespeare and hamlet” results in 4079 citations. A similar query for “joyce and ulysses” returns 3269. Normed for years of industry-building time these figures tell a sad, sad tale for the man from Stratford. Ulysses inspires 48.8 citations per year and Hamlet a meager 10.4.

In this sense, the Bard can be thought of as the steady industrial giant. His stock increases little by little, and he is a generally good investment. For the sake of convenience, let’s call him “GM.”

Joyce, on the other hand is a relative new comer to the marketplace. He is more like a Silicon Valley startup and his stock starts off slow and then sky-rockets. For the sake of convenience, we’ll call him “Google.”

Now, getting back to the central question, who’s bigger. . . You’ll find the answer here.

This week the Chronicle of Higher Education ran an article written by Jennifer Howard about “literary geospaces.” The article featured some work I have done mapping Irish-American literature using Google Earth (and also profiled the work of Janelle Jenstad who has been mapping early modern London).

Photo by Noah Berger

The bit about my Google Earth/Irish-American literature mash up resulted in several emails from folks wanting to know more about the project and more specifics about my findings. . . beware what you ask for. . .

I began building a bibliographic database of Irish-American literature many years ago when I was working on my dissertation (Jockers, Matthew L. “In search of Tir-Na-Nog: Irish and Irish-American Literature in the West.” Southern Illinois University, 1997). In 2002 I received a grant from the Stanford Humanities Laboratory to fund a web project called “The Irish-American West.” At that point I moved the database into MySql and put the whole thing on line with a search interface. As part of the grant, I also began digitizing and putting on line a number of specific Irish-American novels from the west. All of this work was later moved to the web site of the Western Institute of Irish Studies, a non-profit that I helped establish with then Irish Consul Donal Denham and a few other Bay Area enthusiasts. The archive and the database are alive and well at the Institute, and each year students who take my Introduction to Humanities Computing course help the archive grow by encoding one or two more full texts. (The group projects my students complete each year can be found on my courses page)

Ironically, on St. Patrick’s day in 2007, I was invited to present a paper at the 2007 MLA meeting in Chicago as part of a panel session titled “Literary Geospaces.” The paper I delivered “Beyond Boston: Georeferencing Irish-American Literature” utilized Google Earth to help the audience visualize both the landscape and chronology of Irish-American literary history. I warned the audience at the time not to be seduced by the incredible visual appeal of Google Earth; GE is a stunning application, and I was honestly worried that my audience would lose track of my central thesis about the literary history of Irish-America if they got too caught up in the visualization of the data. I was also worried about the amount of time that went into the preparation of the Google Earth mash-up. The MLA is a meeting of literature and language professors, and I didn’t want to give the impression that putting something like this together was a simple matter (along with the Google Earth app itself, I’d utilized php, xml, xsl, html, and Mysql to build the .kml file that runs the whole show).

The central thesis of the paper was that in order to understand Irish-American literature we need to look not simply to the watershed moments of Irish-American history, but we must look to the very geography of America. As long ago as 1997, my research had shown that the Irish experience in America was largely determined by place. It’s true, of course, that the time of immigration to the U.S. was important in coloring the Irish experience: were these pre-famine immigrants, famine refugees, or the 1980’s so-called “commuter Irish.” But I discovered that equally important to chronology was place and the business of where the immigrants settled. For my research, I divided the country up into a number of regions (Midwest, mountain, southwest, pacific. . .) and each one of these regions turned out to have a distinct “brand” of Irish-American writing. Generally speaking, though, the further west we go the more likely we are to find writers describing the Irish-American experience in positive terms. And perhaps more importantly, the further west we go the more Irish writing there seems to be if we view “more” in relative terms, as a percentage of the Irish population.

I suppose one of the most interesting things I discovered along the way involves what was happening in the early part of the 20th century. My colleague Charles Fanning has speculated that in the early 1900s, from around 1900 to 1930, Irish-Americans turned away from writing about their experience in the United States. These were difficult times for Irish-Americans, and Fanning writes in his impressive book The Irish Voice in America how “a number of circumstances–historical, cultural, and political, including the politics of literature–combined to [create] a form of wholesale cultural amnesia (3).”

What I discovered was that Irish writers in the western U.S. were largely undeterred.

And this all made perfectly good sense: Irish writers in the West did not have to face the same prejudice that there counterparts in the East faced. There was no established Anglo-Protestant majority in the West, there was far less competition for good jobs, and generally speaking the Irish who ventured west were better off and typically better educated than their countrymen in the East. Thus they had more means and more opportunity for writing. So if we look at the entire corpus we find not a period of literary recession in the early 1900s, but instead a period of heightened activity. It’s only when we probe that activity that we discover that writers from west of the Mississippi are the ones being active.

Here is a link to a Quicktime video of the Google Earth mash-up. I’m still working on setting up an interactive version that will query my database dynamically and allow visitors to sort and probe the entire collection. . . more on that later.

Recently (4/2008) I had reason to Part-Of-Speech tag a whole mess of novels, around 1200. I installed the Stanford Tagger and ran my first job of 250 novels on an old G4 under my desk. Everything worked fine, but the job took six days. After that experience, I figured out how to utilize xGrid for “distributed” tagging, or what I’ll call, according to convention, “Tagging@Home.” At the time that I was working on this tagging project, the folks in Stanford’s NLP group, especially Chris Manning and Anna Rafferty, were improving the tagger and adding some XML functionality to the program. I’m very grateful to Chris and Anna for their work. What follows is a practical guide for those who might wish to employ the tagger for use with XML or who might want to understand how to set up xGrid to help distribute a big tagging job.

First I provide some simple examples showing how to take advantage of the XML functionality added in the May 19, 2008 release of the
Stanford Log-linear Part-Of-Speech Tagger. Further down in this page I include information about setting up xGrid to farm out a large tagging job to a network of Macs, useful if you want to POS tag a large corpus. These example assume that you have installed the tagger and understand how to invoke the tagger from the cmd line. If you are not yet familiar with the tagger, you should consult the ReadMe.txt file and javadoc that come with it. In the javadoc, see specifically the link for “MaxentTagger” where there is a useful “parameter description” table.

Example One: Tagging XML with the Stanford Log-linear Part-Of-Speech Tagger

Many texts are currently available to us in XML format, and in literary circles the most common flavor of XML is
TEI. In this example we will POS tag a typical TEI encoded XML file.

Begin by examining the structure of your source XML file to determine which XML elements “contain” the content that you wish to POS tag. More than likely, you don’t want to tag the title page, for example, but are interested primarily in the main text. To complete the exercises below, you may want to
download a shortened version of James McHenry’s
The Wildernesswhich is the text I use in the examples; alternatively, you may use your own .xml file. The example file “wilderness.xml” is marked up according to TEI standards and thus contains two major structural divisions: “teiHeader” and “text.” The “teiHeader” element contains metadata about the file and the “text” element contains the actual text that has been marked up. For purposes of this example, I shortened the text to include only the first two chapters.

For this example, I assume that you wish to POS tag the text portions of the book that are contained in the main “body” of the book, that is to say, you are not interested in POS tagging the title page(s) or any ancillary material that may come before or after the primary text. To separate the main body of the text, TEI uses a “body” element, so we might begin by having the tagger focus only on the text that occurs within the body element.

If we were simply tagging a plain text file, such as what you might find at
Project Gutenberg, the usual command (as found in the tagger “readme.txt” file) would be as follows:


java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/bidirectional-wsj-0-18.tagger -textFile sample-input.txt > sample-output.txt


To deal with XML based input, however, we need to add another parameter indicating to the tagger that the source file is XML and more importantly that we only want to tag the contents of a specific XML element (or several specific elements, which is also possible). The additional parameter is “-xmlInput” and the parameter is followed by a space delimited list of XML tags whose content we want the POS tagger to tag. The revised command for tagging just the contents of the “body” element would look like this:


java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/bidirectional-wsj-0-18.tagger -xmlInput body -textFile ./pathToYourDirectory/Wilderness.xml > ./pathToYourDirectory/wilderness-output.xml


When invoked in this manner, the tagger grabs the content of the body tag for processing and ignores (or strips out) any XML markup contained within the selected tag. Running the command above, thus has the effect of stripping out all of the rich markup for chapters, paragraphs and etc. In order to preserve more of the structural markup, a slightly better approach is to use the “p” tag instead of the “body” tag, as follows:


java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/bidirectional-wsj-0-18.tagger -xmlInput p -textFile ./pathToYourDirectory/Wilderness.xml > ./pathToYourDirectory/wilderness-output.xml


If you run this command, you’ll be a bit closer, but you might notice that our source file includes a series of poetic epigraphs at the beginning of chapters and a periodic bit of poetry dispersed elsewhere throughout the prose. Using just the “p” tag above, we fail to POS tag the poetic sections. Fortunately, the tagger allows us to specify more than one POS tag in the command, and we can thus POS tag both “p” tags and “l” tags (which contain lines of poetry)
*see note*:


java -mx300m -classpath stanford-postaggerar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/bidirectional-wsj-0-18.tagger -xmlInput p\ l -textFile ./pathToYourDirectory/Wilderness.xml > ./pathToYourDirectory/wilderness-output.xml


Complete execution time for the command above using McHenry’s
Wildernesson a typical desktop computer is about 30-40 seconds but the processing time will vary depending on your machine and the speed of your processor. Here is a snippet of what the resulting output will look like:


<l>As/IN slow/JJ our/PRP$ ship/NN her/PRP$ foamy/NN track/NN ,/,</l>
<l>Against/IN the/DT wind/NN was/VBD cleaving/VBG ,/,</l>
<l>Her/PRP$ trembling/VBG pendant/JJ still/RB look/VB 'd/NNP back/RB</l>
<l>To/TO that/DT dear/RB isle/VB 'twas/NNS leaving/VBG ;/:</l>
<l>So/RB loth/NN we/PRP part/VBP from/IN all/DT we/PRP love/VBP ,/,</l>
<l>From/IN all/DT the/DT links/NNS that/IN bind/NN us/PRP ,/,</l>
<l>So/RB turn/VB our/PRP$ hearts/NNS where'er/VBP we/PRP rove/VBP ,/,</l>
<l>To/TO those/DT we/PRP 've/VBP left/VBN behind/IN us/PRP !/. Moore/NNP</l>

<p>Let/VB melancholy/JJ spirits/NNS talk/VBP as/IN they/PRP please/VBP concerning/VBG the/DT
degeneracy/NN and/CC increasing/VBG miseries/NNS of/IN mankind/NN ,/, I/PRP will/MD not/RB
believe/VB them/PRP ./. They/PRP have/VBP been/VBN speaking/VBG ill/JJ of/IN themselves/PRP ,/,
and/CC predicting/VBG worse/JJR of/IN their/PRP$ posterity/NN ,/, from/IN time/NN immemorial/JJ ;/:
and/CC yet/RB ,/, in/IN the/DT present/JJ year/NN ,/, 1823/CD ,/, when/WRB ,/, if/IN the/DT one/CD
hundreth/NN part/NN of/IN their/PRP$ gloomy/JJ forebodings/NNS had/VBD been/VBN realized/VBN ,/,
the/DT earth/NN must/MD have/VB become/VBN a/DT Pandemonium/NN ,/, and/CC men/NNS something/NN
worse/JJR than/IN devils/NNS ,/, -LRB-/-LRB- for/IN devils/NNS they/PRP have/VBP been/VBN long/JJ
ago/RB ,/, in/IN the/DT opinion/NN of/IN these/DT charitable/JJ denunciators/NNS ,/, -RRB-/-RRB-
I/PRP am/VBP free/JJ to/TO assert/VB ,/, that/IN we/PRP have/VBP as/IN many/JJ honest/JJ men/NNS ,/,
pretty/RB women/NNS ,/, healthy/JJ children/NNS ,/, cultivated/VBN fields/NNS ,/, convenient/JJ
houses/NNS ,/, elegant/JJ kinds/NNS of/IN furniture/NN ,/, and/CC comfortable/JJ clothes/NNS ,/,
as/IN any/DT generation/NN of/IN our/PRP$ ancestors/NNS ever/RB possessed/VBN ./.</p>


*Note that after the -xmlInput parameter we include the “p” tag and the “l” tag separated by an *escaped* space character. UNIX chokes on the space character if we don’t escape it with a backslash. Martin Holmes pointed out to me that if you are using windows, then you should put the space delimited tags inside quotes (like this: “p l”) and disregard the backslash.

Example Two: POS Tagging an XML source file and outputting results in XML

In addition to being able to “read” xml, the 5-19-2008 tagger release also includes functionality allowing users to output POS tagged results as well-formed xml. The process for doing so is very much the same as Example One above, however, to output XML, we need to add an additional “-xmlOutput” parameter to the command.


java -mx300m -classpath stanford-postaggerar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/bidirectional-wsj-0-18.tagger -xmlInput p\ l -xmlOutput -textFile ./pathToYourDirectory/Wilderness.xml > ./pathToYourDirectory/wilderness-output.xml


In this case, the resulting output is a far richer XML in which each sentence is wrapped in a “sentence” tag and each word in a “word” tag. The sentence tag includes an “id” attribute indicating its number in the sequence of sentences in the entire document. Likewise, the word element contains an id attribute referecning the word’s position in the sentence followed by a “pos” attribute indicating what part of speech the tagger assigned the word. Here is a sample of the output:


<p>
<sentence id="17">
<word id="0" pos="VB">Let</word>
<word id="1" pos="JJ">melancholy</word>
<word id="2" pos="NNS">spirits</word>
<word id="3" pos="VBP">talk</word>
<word id="4" pos="IN">as</word>
<word id="5" pos="PRP">they</word>
<word id="6" pos="VBP">please</word>
<word id="7" pos="VBG">concerning</word>
<word id="8" pos="DT">the</word>
<word id="9" pos="NN">degeneracy</word>
<word id="10" pos="CC">and</word>
<word id="11" pos="VBG">increasing</word>
<word id="12" pos="NNS">miseries</word>
<word id="13" pos="IN">of</word>
<word id="14" pos="NN">mankind</word>
<word id="15" pos=",">,</word>
<word id="16" pos="PRP">I</word>
<word id="17" pos="MD">will</word>
<word id="18" pos="RB">not</word>
<word id="19" pos="VB">believe</word>
<word id="20" pos="PRP">them</word>
<word id="21" pos=".">.</word>
</sentence>
</p>


Using the Stanford Log-linear Part-Of-Speech Tagger with xGrid

There is really nothing special about using the tagger on an xGrid, but since it took me six hours to figure out how to do it and how to set everything up, I provide below a basic set up guide that will have you tagging in ten minutes (give or take an hour)

Not being a sys admin, I found the available documentation about xGrid a bit tricky to navigate, and frankly it just didn’t address what I feel to be rather fundemental questions about exactly how xGrid does what it does. More useful than Apple’s own xGrid documentation were
Charles Parnot’s xGrid tutorials available
here (The Apple manual even refers to these). Especially useful is Parnot’s “GridStuffer” application (more on that in a minute). The one problem I had with Parnot’s tutorials is that they all assumed that I was using a single machine as Client, Controller, and Agent. In xGrid lingo, the Client is the machine that submits a job to the Controller. The Controller is where xGrid “lives,” and it is the Controller that serves as the distributor and the distribution point for sending a job out to the “Agents.” Agents are the machines that are enlisted to do the heavy lifting, that is, they are all the machines on the network that are signed up to parallel process your job.

Example Three: POS Tagging with xGrid from the Command Line

To make life easy, the first thing I did was to figure out how to submit the job to xGrid from the cmd line. In this case I had ssh access to the server hosting the controller, so I installed the Tagger in a folder I created inside /Users/Shared/. The full path to my installation of the tagger was “/Users/Shared/Tagger/stanford-postagger-full-2008-05-19/”. Once this was done, I cd’ed (changed directory) down into this directory and then entered the following cmd at the prompt (you would substitute “hostname” and “password” with your information, e.g. – h myserver.mydomain.edu -p myPW)

xgrid -h <hostname> -p <password> -job submit /usr/bin/java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -arch bidirectional -lang english -model models/bidirectional-wsj-0-18.tagger -textFile sample-input.txt

You will notice one thing about the tagger portion of this cmd that is different: there is no redirect of the output. Above, when we were just tagging files without xGrid, the cmd ended like this “-textFile sample-input.txt > sample-output.txt” telling the program to print its output to a new file titled “sample-output.txt”. xGrid has it’s own way of handling “results” or stdout; xGrid saves the results of a job on its own and then allows us to retrieve the results using a job id.

So, after submitting the cmd above, xGrid returns some important information to the terminal, somehting like this:


{
jobIdentifier = 2822;
}

This tells us that our job is id number 2822, and we’ll need that number in order to “return” the results after the job has finished. But after we submit the job we can also check on its status by entering the following cmd:


xgrid -h <hostname> -p <password> -job attributes -id 2195


Which returns something like this:


{
jobAttributes = {
activeCPUPower = 0;
applicationIdentifier = "com.apple.xgrid.cli";
dateNow = 2008-05-23 15:15:16 -0700;
dateStarted = 2008-05-23 15:14:45 -0700;
dateStopped = 2008-05-23 15:14:49 -0700;
dateSubmitted = 2008-05-23 15:14:41 -0700;
jobStatus = Finished;
name = "/usr/bin/java";
percentDone = 100;
taskCount = 1;
undoneTaskCount = 0;
};
}


jobStatus here tells us if the job is running, if it is finished or if it has failed. This job has finished, and so I enter the next cmd to retrieve the results:


xgrid -h <hostname> -p <password> -job results -id 2195


This command tells the xGrid to print the results of the job to my Terminal window, which isn’t all that great if I’ve just tagged a big file. What I really want is to send the result to a new file, so I can modify the cmd above to redirect the output to a file, like this:


xgrid -h <hostname> -p <password> -job results -id 2195 > sample-output.txt


You’ll now find a nicely POS-Tagged file titled “sample-output.txt” in your working directory.

Example Four: POS Tagging with xGrid Using Charles Parnot’s GridStuffer

You can download a copy of Charles Parnot’s GridStuffer at the
xGrid@Stanford website. GridStuffer is a slick little application that greatly simplifies the work involved in putting together a large xGrid job. You should read
Parnot’s GridStuffer Tutorialto understand what GridStuffer does and how it works. Everything that I do here is more or less exactly what is done in the tutorial, well, not exactly. The real point of difference pertains to the input file that contains all the “commands.” Because the Stanford Tagger is a Java application (and not quite like Parnot’s Fasta program) I had to figure out how to articulate the command lines for the GridStuffer input file. In retrospect it all seems very logical and simple, but trying to move from Parnot’s example to my own case actually proved quite challenging. In fact, it was only by reading through one of the forums that I discovered a key missing piece to my puzzle. Other users had reported problems calling Java application and a common thread had to do with paths and file locations. . .

The real trick (and it’s no trick once you understand how xGrid and GridStuffer work) involves how you define your paths and where you place files. I spent a lot of time trying to figure this all out, and even Parnot admits that understanding which paths are relative and which paths are absolute can get challenging. Let me explain. With GridStuffer, you are not, as I did above, logging into the Controller machine (server) and running an xGrid command locally. Instead, you are running GridStuffer on your own machine, separate from the Controller machine and thus acting as a true “Client.” This makes one wonder, right from the start about whether you need to be concerned about file paths on your machine or on the controller or on both. The answer is, “yes.”

Now there are always many ways to skin the cliche, so don’t assume that the way I set things up is the only approach or even the best approach; it is, however, an approach that works. I began by installing the tagger on my local machine (the machine that would serve as the Client and from which I would launch and invoke GridStuffer. I installed the tagger in my “Shared” directory at the following path: /Users/Shared/tagger/stanford-postagger-full-2008-05-19

Inside of this root folder of the tagger (stanford-postagger-full-2008-05-19), I then added another folder titled “input” into which I copied all of the files that I wanted to tag (in this case several hundred novels marked up in TEI XML). Next I created the “input file” (or “commands”) that GridStuffer requires and most importantly, I
put it into the exact same directory(stanford-postagger-full-2008-05-19). Well, the truth is that I did not actually do this at first and spent a good deal of time trying to figure out why the program was choking. In Parnot’s tutorial, he has you store these files on a folder on your Desktop. This practice apparently works just fine with his Fasta tutorial, but it makes a java app like the POS Tagger (and others) choke. Moving the input file to the root directory of the Tagger application solved all my problems (well, almost all of them). Anyhow, not only is the placement of this file important, but this is the critical file in the entire
shebang; it is the file with *all* the calls to the tagger. Here is a snippet from my file:


/usr/bin/java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -arch bidirectional -lang english -model models/bidirectional-wsj-0-18.tagger -xmlInput p -textFile input/book.3.xml
/usr/bin/java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -arch bidirectional -lang english -model models/bidirectional-wsj-0-18.tagger -xmlInput p -textFile input/book.4.xml
/usr/bin/java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -arch bidirectional -lang english -model models/bidirectional-wsj-0-18.tagger -xmlInput p -textFile input/book.5.xml
/usr/bin/java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -arch bidirectional -lang english -model models/bidirectional-wsj-0-18.tagger -xmlInput p -textFile input/book.6.xml


Now for my big job of several hundred novels, I didn’t actaully cut and paste each of these. I wrote a simple script to iterate through a directory of files and then write this “commands” file. I used a little php script run from the command line of the terminal; you could use Perl or Python or some other for the same purpose. Note in the code above that I am tagging xml files and selecting the contents of the “p” elements just as I did in the tagger examples above.

It took me about six bangs of my head on the desk to figure out that I needed to provide the full path to “java” (i.e. /usr/bin/java). That was the only other counterintuitive bit. With this as my input file, I then selected an output directory using the handy Gridstuffer interface and Voila! I was soon tagging away.

This is worth a look for those thinking about online collaboration: http://www.sophieproject.org/